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INTJfdDUCTION ' •***.■*' 



I 



I . INTRODUCTION 



The purpose of this study was to evaluate the quality and utility of NCEC 
information analysis products, including ERIC clearinghouse products, PREP 
reports, and EMC bibliographies.* This project was supported by the Office 
of Program Planning and Evaluation in the U.S. Office of Education (USOE) , 
and was conducted over a 12-month period from July 1971 through June 1972. 



As stipulated by USOE, the study was to focus only on the products, and not 
on the management process by which they were conceived and prepared. Within 
this limitation of scope, the planning and conduct of the study were guided 
by two major goals: 

• To develop data from a cross-section of educators 
regarding their level of familiarity with, and 

- judgments on the quality and utility of, NCEC 

information analysis products. Although based on 
specific documents, the data would be analyzed in 
relation to characteristics of both the user 
population and the documents so the results could 
assist USOE in developing policy-related guide- 
lines for their future information analysis 
activities. 

• To assess the SDC survey methodology so that a 
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A. NCEC INFORMATION ANALYSIS PRODUCTS 

The literature of research and practice is synthesized in three major types of 
NCEC information analysis products: ERIC clearinghouse products, EMC 

bibliographies, and PREP reports. A brief background on each of these product 
groups is provided in the following sections. 



1 . 



ERIC CLEARINGHOUSE PRODUCTS 



Each of the 19 ERIC clearinghouses acquires, screens, indexes, and abstracts 
the published and unpublished literature in its respective content area. The 
products of these efforts are published in Research in Education and Current 
Index to Journals in Education . In addition, the clearinghouses produce a 
number of special information analysis products on subjects relevant to their 
scope of coverage. These products represent not only a range of product types 
(e.g., bibliographies and reviews), but a number of formats (e.g., newsletters 
and monographs) and dissemination media (e.g., journal articles and chapters 
in books). Adequate definition of "ERIC information analysis product," 
therefore, was a challenging part of the initial project work. Through a 
process of analysis and refinement, a decision was made to include three major 
types of products: bibliographies (citations only, citations with abstracts, 

and citations with annotations), reviews and state-of-the-art papers, and 
practical guidance papers. 

The various special information analysis products have been prepared by the 
ERIC clearinghouses for the past 6 years. The steady growth of this program 
is illustrated in the yearly increments of publications cited in ERIC Products , 

< \y/ . f ' 

an annual, bibliographic publication of the ERIC Clearinghouse on Library and 

^v\ 

Information Sciences. These figures are: \ 

'*>'"1967-1968 149 citations 



1968- 1969 

1969- 1970 

1970- 1971 



240 citations 
366 citations 
416 citations 



Although some products'are channeled through the professional journal literature, 

most of the products under study were originally available as monographs, for 

$ 
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which initial press runs were made for clearinghouse distribution, followed by 
distribution through the ERIC Document Reproduction Service (EDRS) . 

Each year, the clearinghouses submit budget plans to NCEC that outline the 
number and types of information analysis products proposed for the ensuing year. 
Approximately 40 to 45 percent of the total budget for each clearinghouse is 
for the information analysis program. The total NCEC budget for this program 
is approximately one million dollars. 



2. PREP REPORTS 

The Targeted Communications program of the NCEC's Division of Educational 
' Extension Systems (formerly the Division of Practice Improvement) is the 
foundation for interpretive summaries called PREP reports. Projects are funded 
through contracts and grants to interpret research and development findings 
that have a potential for Improving educational practice, particulcrly in USOE 
priority areas. The progrsm specifies that the projects should be described 
in non-technical language and in such a way that they will meet the needs of 
specific, non-research audiences. 

The actual PREP ^reports are created within NCEC as a byproduct of the Targeted 
Communications program. (The PREP report budget is a small fraction of the total 
budget for the Targeted Communication program.) Although these reports use 
much of the material— verbatim— from the final project reports, some formal 
or organizational changes are sometimes made. For example, a given project 
report may be repackaged into more than one PREP report. 

To date, 30 PREP reports have been prepared. Approximately 300 copies of each 
are sent by USOE to state education agencies, who in turn have primary 
responsibility for distribution to appropriate target audiences. These reports 
are also made available through the U.S. Government Printing Office and the 
ERIC Document Reproduction Service. 

Workshops and conferences for disseminating information are often a part of 
Targeted Communication projects. Originally, several workshops were to have 
been evaluated in this study. However, investigation revealed that evaluations 



had already been conducted on most of these workshops, by the sponsors, 
immediately following the workshops. Moreover, since considerable time had 
elapsed since the workshops had occurred, there was serious doubt that the 
participants would remember the workshop content or be able to identify its 
impact . 



3. EMC BIBLIOGRAPHIES 

Bibliographic reporting services provided by the Educational Materials Center 
(now a part of the expanded Educational Reference Center) draw upon a 
collection of approximately 16,000 textbooks, children's books, and professional 
education materials provided by publishers on a "permanent loan" basis. Most 
of this collection is housed at the Federal City College in Washington, D.C. 

Over a period of 11 years, from the time it was the Educational Materials 
Laboratory, the Center has prepared 54 bibliographies. Although changes in 
formats have occurred over the years, the purpose has always been the same: 
to provide educators with bibliographic information on the particular materials 
in the Center's collection. 

EMC bibliographies are distributed free by USOE and sold through the U.S. 
Government Printing Office. 




! 
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B. 



ISSUES TO BE ADDRESSED IN THE STUDY 



In its Request for Proposals, USOE identified a number of issue.- that needed to 
be addressed in the study. For each product in the sample, the study was to 
attempt to answer questions such as the following: 

• Is the intended audience aware of the product s existence? Ho* 
many have read it? 

• How was it received by its intended audience? 

• What is the quality of the product as perceived by. intended users 
and others qualified to judge? What were its strengths and 
deficiencies? 

• Did it accurately summarize and defensibly interpret the 
relevant literature? 

• Do the bibliographies direct the user to the appropriate 
literature? 

• How useful was the product to intended users? What use did they 
make of it (e.g., was it considered general information; was 

it used in solving a specific educational problem)? 

• Did it satisfy user needs? What needs? 

• Did the product reflect old, current, or original approaches 
to problems? Are the problems timely? 

• What impacts can users report on practice as a result (even 
a partial one) of reading the publication? 

• How do qualified experts rate the report as a contribution to 
information distillation of the educational literature? 

• What were the ''side—ef fects 1 of the document (e.g. , use by 
persons other than the intended audience, utility other than 

intended, etc.)? 

Each of these, and other related questions, were carefully considered in the 
design of the survey plan, particularly in the selection of survey participants 
and the development of survey instruments. Features of the SDC approach are 
summarized in the next section. 



c. 



STUDY APPROACH 



The overall plan for the study was to obtain an appropriately large number of 
quality-utility judgments on NCEC products from a sample representative of the 
educational community. Features of the study plan are highlighted below: 

• The Product Sample. A fairly large sample (146) of NCEC products 
was carefully selected to represent the major kinds of products 
and the major content areas. 

0 The Two Surveys . Two surveys were conducted: 

- A General Field Survey, representing a broad cross- 
section of educators, and 

- A Specialists’ Survey involving individuals identified by 
their colleagues as being particularly well qualified to 
evaluate documents in the product sample. 

Participants for the General Field Survey were drawn in two 
ways : 

- a rigorous samp le— — the random sample— - was drawn from 

, personnel listings of state education agencies, local 

school districts, junior colleges, colleges/universities, 
and USOE-support^d research facilities. 

— A second sample— — the non— random sample - —was drawn from 
several available listings, including ERIC Clearinghouse 
mailing lists, state and local educationax information 
center user lists, and ERIC Document Reproduction Service 
on-demand sales records. 

• The Surv ey Instruments . Four ques tioimaires were developed for 
the stray ' 



- A Screening Questionnaire ( Q 1 > was mailed to participants 
In The General Field Survey to identify educators familiar 
with NCEC products. It included a special color insert 
of miniature photos of sample products. 




1-6 



t 



e- O 

A. U 



% 






- A User Evaluation Questionnaire (Q2) was mailed to a ^ 
selected group of respondents to the Screening Questionnaire 
who had read or skimmed products from at least one NCEC 
unit. Participants were asked to evaluate 10 documents, 
each of which was individually assigned on the basis of 
Screening Questionnaire data. A document representation 
(i.e., title page, table of contents, and abstract or 
extract) was attached to each questionnaire. 

- A Non-user Evaluation Questionnaire (QSX^was mailed to a 
selected group of - respondents to the Screeiiing Questionnaire 
who reported having not read or skimmed NCEC products. 

An abbreviated form of the User Evaluation Questionnaire 
was developed to explore non— users potential interest in the 
documents. Procedures and packages comparable to those 
of the User Evaluation Questionnaire were used. 

- A Specialists [ Questionnaire (Q4) was mailed to selected 
specialists. Some ’questions were comparable to those of 
the User Evaluation 'Questionnaire, but explored the 
quality dimension in more depth. Documents were 
individually assigned and each specialist evaluated an 
average of two or three documents. Complete copies of 
documents were provided. 

• Data Analysis . Data from the four questionnaires were analyzed 
to relate to each of the study issues and questions,.-. 

Several kinds of survey findings are reported:- 

- Evaluation data from Readers* Non-Readers, / 4hd x ^pecialists 
are displayed in individual document evaluation profiles. 

- Evaluation data from Readers are aggregated on documents 
for each product type, subject area, and user group, 

as well as by level of product exposure and level of 
f effort involved in the production of the product. 

^ — Non-reader and Non— user data are reported in the 
aggregate for documents in each product group. 

- Specialists data are reported for individual documents 
and, in the aggregate, for each of the three product 
types. 



These and other special analyses are reported in one of two volumes of this 
report, as discussed in the next section on the organization of tn:J- report. 
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ORGANIZATION OF THE REPORT 



This study is reported In two volumes. Volume I describes the study objectives, 
reviews the survey methodology, and reports general findings and conclusions. 

As a supplement, Volume II contains the basic evaluatior data, from Specialists, 
Readers, and Non-readers, on each of the 146 documents in the product sample. 

The next chapter in Volume I is an executive summary that is written to 
provide an overview of the entire study in capsule form. Chapter III presents 
a detailed account of the survey methodology, from the development of the 
product sample through the conduct of the General Field and Specialists’ 

Surveys • In Ch&pter IV, the respondent populations of these surveys are 
described by their various user characteristics. 

Chapter V begins the reporting on survey findings with a presentation and 
discussion of data regarding the respondent populations’ levels both 
general and product-specific— of familiarity with NCEC information analysis 
products. A comparison of the two samples of the General Field Survey is made 
in Chapter VI, paving the way for the report in Chapter VII on the several 
aggregated data analyses. These analyses, and the conclusions and recommendations 
in Chapter VIII, are organized by study objectives and issues introduced 

in this Chapter. 

Supplementary materials and tables are contained in several Appendices at the 
end of this volume. Appendix G is a glossary of terms used throughout this 
report. - v 
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THE PRODUCT SAMPLE 



A representative sample of 146 NCEC products was selected from a product universe 
of approximately 500 documents prepared during the period from July 1969 through 
December 1970. Although information was available on the size and general nature 
of this product universe, no systematically gathered and structured data existed 
for determining precise product sampling requirements. An estploratory sampling . 
plan was developed and applied to the analysis and classification of the products 
along several dimensions, including product type and subject matter. A stratified 
random sample was selected to include a greater number of products — Reviews and 
Practical Guidance Papers — that presumably require a- relatively high level of 
effort. Cost-related data and visibility (product exposure) data on each product 
were then obtained from NCEC units for several special analyses. 




2. THE SURVEY POPULATIONS 

Two surveys were conducted: 1) a General Field Survey of a broad cross -section 

of educators and 2) a Specialists* Survey of individuals identified by .heir 
colleagues as being especially well qualified to provide in-depth evaluations of 
particular groups of documents in the product sample. Participants in the ^3 
General Field Survey were drawn in two ways: 

. • A rigorous sample of 1,588 educators— the random sample — was 

drawn from personnel listings of state education agencies, 
local school districts, junior colleges " coJl^eges and univer- 
sities, and USOE-supported research facilities^. 

• A second sample of 3,221 educators— the non-random sample — wa~. 
drawn from several available lists of educators, including 
ERIC clearinghouse mailing lists, state and. local educational 
information center user lists, and ERIC Document Reproduction 
Service (EDRS) on-demand sales records. 

\ 

% 

This dual approach in sampling was used to address two evaluation objectives: 

1) assessing the outreach of documents apd 2) evaluating their quality and 
utility. The random sample could best test the outreach objective, .whereas 
the non-random sample would most likely be necessary to identify product eval- 
uators. It was also important to obtain some comparative data on the two 
sampling approaches so that recommendations could be made for identifying a 
representative group of product evaluators in future evaluations. 
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II. SUMMARY 



A. PURPOSE AND SCOPE OF STUDY 




The purpose of this study was to assist the U.S. Office of Education in develop- 
ing policy-related guidelines for the future development and dissemination of 
NCEC information analysis products, by evaluating the quality and utility of 



these products. 

1 

The study was concerned with three types of information analysis products: 

• Reviews of research and practice and state-of-the-art papers 

from ERIC (Educational Resources Information Center) clearing- 
houses , - — - * - • • - - - - 

, 4 

• Practical Guidance Papers, including PREP (Putting Research 
into Educational Practice) reports and reports from ERIC 
clearinghouses 

t Bibliographies, including those from EMC (Educational Materials 
Center) and ERIC clearinghouses 

As stipulated by USOE, the project was to focus only on the products and not on 
the means by which they were conceived and prepared. Within this scope, the 
study had two major goals: 

• To obtain information from a cross-section of educators regard- 
ing their level of familiarity with NCEC information analysis 
products and their judgments of the quality and utility of those 
products. 

• To assess the survey methodology ,4- n terms of its potential 
applicability to continuing evaluations of NCEC products. 



B. STUDY DESIGN 

The study was designed to address several specific topics, including the ways in 
which these products are used, the needs they meet, the degree of user satisfac- 
tion with their quality and utility, and their impact on educational practice. 

\ 

Key feature's of the study design are summarized below. 



The 194 Specialists were selected on the basis of recommendations from ERIC 

4 

Clearinghouse advisory board members. 



3. THE SURVEY INSTRUMENTS 




The study required the development of four instruments; 

• A short Screening Questionnaire (Ql) was designed to identify 
educators who were users of NCEC information analysis products 
and were therefore potential evaluators of specific products. 

It also Included items designed to obtain descriptive informa 
tion for characterizing respondents by work setting, role/ . 
function, and professional areas of interest so that documents 
could be assigned for the evaluation survey. This questionnaire 
contained a color insert displaying, in miniature-sized color 

. photos, examples q£_NCEC products. ... 

Respondents to this questionnaire were classified as Users if 
they Indicated having read/skimmed products from at least one 
NCEC unit (e.g., from an ERIC clearinghouse); they were Non- 
users if they reported no awareness of products or had only 
heard of products. 

m 

• a User Evaluation Questionnaire (Q2) was designed in two sec- 
tions. The major group of items was developed to obtain from 
Users that had previously read specific documents evaluations 
of their quality, utility, and impact. A brief section was 
included to obtain evaluations of Users* potential interest in 
products (out of the 10 assigned) that they had not previously 

read. 

This questionnaire had two versions — one for Reviews and Prac- 
tical Guidance Papers and the other for Bibliographies. For 
each document, a recall aid in the form of a one-page represen- 
tation consisting of title page, table of contents, and abstract 

or extract was attached. 

* 

All respondents to this questionnaire were classified as Users. 
However, in terms of their use of the 10 particular products 
assigned to them for review, each respondent became: 1) a Reader 
in cases where he had previously read/skimmed a particular 
product and/or 2) a Non-reader in cases where he had only 
heard about a product or had no prior famliarity with it. 

• A Non-User Evaluation Questionnaire (Q3) was deigned to obtain 
evaluations of potential interest from individuals who were not 
familiar with NCEC products. This questionnaire also included 
a one-page representation of each of the 10 documents a Non- 
user was to evaluate. 



These questions were comparable to those asked of the Non- 
readers in Q2. Participants were requested to judge the pro- 
ducts * potential relevance to their professional interests; 
the potential usefulness (in terms of the participants’ needs 
for a product); and, for those participants who had previously 
heard about a product, to indicate the reasons for non— use* 

Respondents to this questionnaire were classified as Non-users . 

• A Specialists’ Evaluation Questionnaire (Q4) was designed to 
obtain in-depth evaluations of the quality and utility of docu- 
ments that were in the Specialists’ areas of expertise. This 
questionnaire, which was accompanied by a copy of the full 
document to be evaluated, provided for free-form, as well as 
structured, responses. Specialists were asked to review pro- 
ducts in terms of the needs of the profession, in general, and 
not in terms of their personal needs and uses. 



4. THE SURVEY PROCEDURES 

The General Field Survey involved the following major steps: 

• Screening Questionnaires (Ql) were mailed to the entire sample 
(both the random and non-random groups). In all, 4,692 Qls 
were mailed and the return rate, with one followup mailing, 
was 64 percent * 

Although college and university personnel were somewhat over- 
represented, in terms of role and function, this population (from 
both samples) was representative of the various educational user 
groups. Teachers /Professors had the highest representation, 
followed by Program Specialists/Consultants and Librarians/ 

Media Specialists. 

• Those Ql respondents who indicated that they had previously 
read NCEC products were considered candidates for the User 
Evaluation Questionnaire (Q2) . A total of 1,837 product users 
identified through Ql were sent Q2 and asked to evaluate 10 
Individually assigned products. Returns to Q2 numbered 1,251, 
or 68 percent, and included fairly comparable percentages 

of user groups from both samples. 

• Those respondents who reported not having previously read 
products from any NCEC unit were sent Non-User Evaluation 
Questionnaires (Q3) . The assignments of 10 products each 
were made individually on the basis of interests reflected 
in Ql returns. Returns to Q3 numbered 255, or 71 percent. 
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The Specialists ' Survey was carried out separately from the 
General Field Surrey. The 194 Specialists provided ia-depth 
evaluations on all products: for 127 products, it was poss 

to obtain evaluations from three different Specialists; for t 
remainder, two evaluations were obtained. 



- 'Document assignments to Users were made individually on the basis of information 
provided in their Q1 returns: 1) work setting, 2) role/function, 3) major and 

other areas of professional interest, and 4> NCEC units (e.g., an ERIC clearing- 
house or product group for which they reported previous use). Non-users were 
also assigned products individually in the same way. Assignments to Specialists 
were made on the basis of recommendations '‘made by the ERIC clearinghouse advisory 
board members and of information provided by the Specialists in telephone con- 
venations with project staff members. - • - 



5. DATA ANALYSIS 

Data from the four questionnaires were analyzed for relation to the major study 
issues and questions. Preliminary analyses were performed on Q1 to study the 
respondent population and particularly its familiarity with NCEC products. The 
Q1 analysis showed that, in comparison with the random sample, there was a 
greater representation of college and university personnel in the non-random 
sample, as well as greater awareness of NCEC products and a greater breadth of 
interest. A more comprehensive analysis of the two sample groups was performed 
on Q2 Reader evaluation data to help provide a basis for deciding ^whether to 
combine data from the two samples for further analyses. There was no pattern 
of statistically significant differences in their evaluation data. Therefore, 
the two samples were combined for further analyses of Q2. 



Questionnaire items were related to the overall dimensions of quality, utility, 
and impact so that data from each evaluator group could be analyzed for individual 
products and for special aggregations of data. These specific elements of 
quality and utility were studied separately and not combined into a single 
quality or -utility index. These elements included. 




II-5 



• -> 



• Quality ; Coverage, up-to-dateness, format, clarity of writing, 

thoughtfulness of discussions, etc. 

• Utility ; Use and degree of usefulness, comparative usefulness 

(with other products of the same type) , relevance, 
and need. 

Only Readers, as general users, were asked to indicate the impact of information, 
i.e., how it was used in decision making and in their daily work. 

. * 

Mean scores and percentages were computed on responses from Readers, Non-readers, 
and Non-users; data from Specialists were tabulated for each document and per- 
centages were used in aggregate analyses. 

Data from Readers, Non-rdlders , and Specialists were analyzed for each product 
and are reported in Volume XX of this report. Special aggregate analyses were* 
performed on the basic grouping of documents by product type. Further analyses on 
this grouping were performed by user group (e.g. , Elementary Teachers and 
Superintendents), subject area (e.g., Instructional Content and Educational 
Administration and Services), and by levels of effort and visibility. 

Most of the findings reported below are from Reader and Specialists* data. 

Non-user and Non-reader data were limited to three areas; potential relevance, 
potential need, and reasons for not reading. 

C. MAJOR SURVEY FINDINGS 

Although the Screening Questionnaire was designed primarily to identify potential 
product evaluators, some preliminary analyses were performed to report the gen- 
eral familiarity of the respondent population with NCEC information analysis 
products. Among the 3,013 respondents to Ql, 87 percent reported familiarity 
with products (i.e., had read/skimmed or were aware of products) from at least 
one NCEC unit. Although the non-random sample, as expected, had a greater 
awareness (94 percent), 72 percent of the random sample reported prior 
familiarity. 
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In terra ol familiarity of the Q2 respondent population with specific P rodu «*' 
approximately 60 percent of the 1,251 respondents had read at least one pro u 
1 Of the 10 assigned; the remainder had read none. (This percentage repr f - 
sentation is composed equally of random and non-random sample partic pan s. 
in 69 percent of the cases (i.e., all potential user-document contacts or 
1,251 respondents times 10), the products had not been heard of or used. 

Non-readers who had previously heard about products reported, in approximately 
30 percent of the cases, that they could not readily obtain copies and, in 
1 percent of the -sees. that they were not sufficiently -interested to do so. 

L the other hand, Non-users who had previously heard about products repor 
that they could not readily obtain copies, in 62 percent of the cases o Pr . 
Guidance Papers and Bibliographies and in 68 percent.of the cases for Reviews 

.. that thev were not sufficiently interested to 
In no cases did Non-users rep, -t that they were no 

seek a copy of the product. 

1. HOW WELL ARE NCEC PRODUCTS RECEIVED BY USERS? 

Quality-related items for Readers included questions on the adequacy of coverage 
of topics, thoughtfulness of discussions, adequacy of organization, c ar y o 
writing, helpfulness of formats, usefulness of textual materials (e.g., 

-* — -w •< " i -“ th 

ments or the number of references in Bibliographies. The overall ratings on 
these quality dimensions were, in most cases, better than the value expresse ^ 
at the mid-point of the scales, indicating definitely positive evaluations. 

The ratings for each product type are strikingly similar. 

t 

The aggregate analyses by user group do not clearly demonstrate that any partic- 

„ r „ l ... i». 

relative differences in judgments on certain quality-related areas. „so 
sample of Elementary Teachers, for example, is often on the extremes of t e 

distribution of ratings. 
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Specialists provided additional coverage of quality-related items, with responses 
to items on accuracy of reporting facts and events; selection of authors; and 
originality and/or thoughtfulness of interpretations. In better than 50 percent 
of the cases. Specialists rated the products in rhe good/excellent categories. 

c 

Comments submitted by both Readers and Specialists indicate a diversity of 
expectations regarding the products— for example, on the selection of materials 
for inclusion (or exclusion) in Bibliographies; on the analysis and interpreta- 
tion sections of Reviews and Bibliographies; and on format and level of treat- 
ment of Practical Guidance Papers. 

A 

■ I \ 

2. HOW USEFUL ARE NCEC PRODUCTS? N. 

Several questionnaire items relate to this particular /study question; they 
include items on the degree of need th^t users hav^xor products, the ways in 
which they are used, and the comparative us efu lness of products. 

• NCEC products are definitely meeting the Reader group’s needs 
for information oi continuing importance to them. To a lesser 
extent, these products §re also meeting their more urgent 
needs. Specialists indicate, even more strongly, that 
products are meeting urgent needs of the field . 

• Both Non-users and Non-readers report, in over 50 percent of 
the cases, that products brought to their attention in the ^ 
survey would probably have been of ’’some use” or "very useful. 

• In general. Readers indicate that Reviews and Practical 
Guidance Papers are used primarily to ’’obtain overviews of ^ 
topics” and ”to update knowledge about already known subjects. 
One of the least frequent uses of these products was "to 
obtain new knowledge." 

However, use varies to some extent with the educator’s role 
and setting. For example, in contrast with the overall pattern 
of use, Elementary Teachers report high usage of Reviews both 
to update knovrledge and to obtain new knowledge. 

• Readers use Bibliographies: first, to identify literature on 

a particular topic; second, to see what kind of new work is 
being reported; and third, to perform comprehensive searches 
of the literature. 



• Specialists report, in over 50 percent of cases, that Reviews 
and Practical Guidance Papers would be ."very useful for most 
uses, including obtaining new knowledge. 

Readers report that NCEC products in the sample are close to very useful in 
comparison with other products of the same type. Bibliographies, with a mean 
score of 2.7 on a 3.0 scale, were rated the highest. Specialists report that 
°evicus are closer to ’’very useful” in comparison to other products of the same 
type, and that Practical Guidance Papers and Bibliographies are slightly less 
so, but still 1 ’worth having available.” 

3. WHAT KINDS OF IMPACT ARE NCEC PRODUCTS HAVING? 

The major' impact of the products is in the application of information to the 
users’ work; they are presumably contributing to educators’ professional growth 
and development. But, beyond this general and rather expected impact, it would 
appear that NCEC products are having two other kinds of impact: 1) a specific 

problem-solving impact and 2) an informational impact. 

• Readers report that NCEC products have been useful in a 
number of* specific problem-solving situations, including 
planning activities, research design efforts, course work. 

(as professors or students), curriculum design, and class- 
room applications with students. 

• Ln approximately half of the cases. Readers report acting as 
channels for the further dissemination of Reviews and Practical 
Guidance Papers v* ' 

• Users report that the Bibliographies and Reviews guide them to 
other products. Users of Bibliographies report a high degree 
of satisfaction with the usefulness of citations in leading 
them to the desired materials. 

4. DO PRODUCTS INVOLVING HIGHER PRODUCTION EFFORT RECEIVE HIGHER RATINGS? 

There was no strong pattern of cprrelation between .^high-effort products and 
high-quality ratings. (The level-of-ef fort measure was professional man-hours 
used in preparing materials.) There is, however, an indication that other 
elements in the production cycle, from conception through dissemination, may 
contribute more significantly to quality ratings. There is, for example, some 



* 

evidence that high-visibility products have higher quality ratings. This may 
suggest that planned visibility (or product exposure) affects quality in that 
the knowledge that a product will have high visibility may place pressures for 
excellence on the entire production effort. 

Although we were not able to pursue the question in this study, there is some 
indication that quality and utility may be fairly independent variables, so 
that a strong need for a product on a given topic outweighs considerations of 
quality. 



D. 



GENERAL CONCLUSIONS AND RECOMMENDATIONS 



This study has demonstrated that NCEC products nre — in varying degrees — known 
and read and that on the whole, they are favorably received by the survey 
respondent populations. It has also shown, however, that the products are 
under-utilized, in part because^, of lack of awareness of the products f existence 

and in part because of a belief that the products are not readily accessible. 

/ 

J 

In ccnj unction with the strong evidence that th^ products actually used are 
meeting important needs and that the potential \ialue to present Non-readers and 
Non-users is great, these findings suggest that: 

• An improved alerting or announcement system — perhaps even a 
selective dissemination of information (SDI) system — needs to 
be developed. 

• The product delivery system needs to be improved and/or an 
intensified education program of how to obtain products needs 
to be developed. 

% 

No generally agreed-upon and tested framework presently exists for judging the 
adequacy of information exchange ^tfmong educators or between information-dissem- 
ination units. The survey findings can be very useful in establishing both 
qualitative and quantitative goals for the growth of the information analysis 
program as well as for standards of excellence for future products. Therefore, 
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we recommend that NCEC and the product developers use the Individual document 
evaluations developed in this survey, with the documents at hand, to: 

• Identify subject areas that users indicate should be updated 
periodically; 

• Study exemplary and deficient products and identify factors 
contributing to their ratings to prepare quality-control 
checklists and develop more complete level-of-ef f ort data. 

Although there were no patterns of differences among the ratings of different 
product types, the types of expectations reflected in evaluators comments and 
the differences in patterns of use among user groups suggest that information 
analysis products need to be redefined, for example, in terms of families of 
Reviews for different uses and different audiences. 

/ 

Since no one product can be expected to meet the diversity of expectations, it 
is also important that each product contain a clear statement of its purpose, 
limitations, and intended audiences. To the extent possible, this information 
should be incorporated into the descriptor system. 

On the basis of this study, we recommend that two separate evaluation studies 
be conducted in the future: 

• An assessment of the outreach of products — using both random 
and non— random samples — should be made periodically , particu- 
larly after the implementation of an improved announcement 
system or of modifications to the delivery system. 

• A continuing evaluation program should be instituted, to 
provide the originating units with current feedback, on NCEC 
products. 

The responsiveness of both the Specialists and the Users suggests that both 
groups are valuable resources that can and should be tapped in future evalua- 
tion studies. Specialists could be used in a regular post-publication review 
system, which could in itself help to announce the products in the professional 
literature. In addition, a general survey — similar to the one used in the 
present study — could be carried out with known product users at some regular 
intervals following the release of products through the various educational 
- dissemination channels. 

II- 11 

r | 




V* 



As the NCEC information analysis program continues to develop and improve, there 
will be an even greater need to detect shifts in the needs and expectations of 
educators and to take the steps necessary to ensure that NCEC products are 
responsive to these needs. The present study has provided a useful start on 
this continuing evaluation. 
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SURVEY METHODOLOGY 



III. REVIEW OF SURVEY METHODOLOGY 



A. OVERVIEW OF SURVEY PLAN 

This chapter describes the preparation for and conduct of two surveys; the 
General Field Survey and the Specialists 1 Survey. ^Five major elements of tht 
survey plan are discussed; 1) The Product Sample; 2) The Survey Instruments; 
j) The Survey Participants; 4) The General Field Survey; and 5) The Specialists’ 
Survey. An overview of each of these activities follows. 

1. THE PRODUCT SAMPLE 

A sample of 146 documents was selected for inclusion in this study. Jhis 
sample includes; 

• 7 PRFP reports 

• 4 EMC bibliographies 

• 135 ERIC clearinghouse products 

IT 

The selection of this sample involved four major steps: identify a product 

universe, analyze and classify these products, reduce the product universe, 
and characterize the final product sample for purposes of analysis. A 1 rig- 
orous sampling procedure could not be applied, for no previously gathered 
data existed from which a universe could be characterized and specific 
parameters established. Instead, the sample was selected through several 
iterations. Each characterization of the documents — by product type, subject, 
level of effort > level of visibility, and intended audiences— provided further 
insights into the composition and makeup of the universe, thereby suggesting 
further requirements for the sampling. 

2. THE SURVEY INSTRUMENTS 

Four questionnaires were developed for use in the surveys. The purpose and 
features of each are shown in Table IXX-1* 
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TABLE III-I. GENERAL DESCRIPTION OF FOUR INSTRUMENTS 
USED IN THE TOO SURVEYS 



1. THE GENERAL FIELD SURVEY 


QUESTIONNAIRE 

Screening 

Questionnaire 

(qi> 


PURPOSES 

• To identify educators 
familiar with NCEC 
information analysis 
products# 

• To obtain descriptive 
information about res- 
pondents for character- 
izing the respondent 
population and assigning 
documents for evaluation. 


FEATURES 

• 4 items 

• A color insert that dis- 
played, in miniature form, 
examples of products from 
NCEC originating unit . 


User Evaluation 
Questionnaire 

CQ2) 

[One version 
(Q2-A) was used 
for Reviews and 
Guidance Papers, 
another (Q2-B) 
for Biblio- 
graphies . ] 


• To obtain from each 
participant quality and 
utility evaluations and 
impact information on 10 
specific documents. 

• To obtain potential- 
interest evaluations of 
documents not previously 
read. 


• 13 items for readers 'of 
specific documents; 2 or 
3, for non— readers. 

• A document representation 
Cone-page display of title 
page, table of contents, 
and abstract or extract) 
for each of 10 documents. 


Non-User 

Evaluation 

Questionnaire 

Ml 


• To obtain potential- 
interest evaluations on 
10 specific documents. 


• 2 or 3 items 

• A document representation 
(one-page display of title 
page, table of contents, 
and abstract or extract) 
for each of 10 documents. 



* • ' 


2. THE SPECIALISTS* SURVEY 


Specialists* 


• To obtain in-depth 


t 7 items, with 10 specific 


Evaluation 


evaluations of documents 


elements of quality 


Questionnaire 


in specialists' areas of 


addressed in a combined 


Ml 


expertise, based on 


rating and free— form 


reading of complete 
documents . 


response format. 



The Screening Questionnaire (Ql) was mailed to a total of 4,692 educators. 

From responses to a question on prior familiarity with NCEC information prod 
ucts , respondents were classed as users or non-users. 

Users received the User Evaluation Questionnaire (Q2) , and were asked to evalu- 
ate 10 different documents. Since the cost of providing full copies was 
prohibitive, document representations were attached to each questionnaire, as 
; i aid to recognition. These representations were one-page displays of docu- 
ment elements, such as title pages, distinctive logos, tables of contents, and 
abstracts (from Research In Education) or extracts (or sample bibliography 
entries) . Two versions of this User Evaluation Questionnaire (Q2) were devel- 
oped, one for substantive papers (Reviews and Practical Guidance Papers) and 
the other for bibliographies. 

An abbreviated form (Q3) of the User Evaluation Questionnaire was designed for 
non-users to explore their potential interest in the document. The questions 
covered potential relevance and usefulness, and — for the non-user who had pre- 
viously heard about the document — reasons why the document had not been read. 

The final questionnaire CQ4) was designed for use in the Specialists* Survey. 
Although many of the questions were comparable to the User Evaluation Question- 
naire, they probed areas of quality more in-depth and used structured 
rating scales in combination with open-ended response formats. Specialists 
were provided with complete copies of the document they were to evaluate. 

s 

Complete questionnaire packages are shown in Appendix D. 

3. THE SURVEY PARTICIPANTS 

Participants for the General Field Survey were identified through a dual samp- 
ling approach. The sources for two samples, which we will refer to as the 
random and non-random samples, are shown in Table III-2. This dual approach 



was used: 



V 



TABLE III-2. GENERAL DESCRIPTION OF SOURCES USED ON TWO 
SAMPLES OF THE GENERAL FIELD SURVEY 



SAMPLE DESCRIPTION 


SOURCES 


A. RANDOM SAMPLE: 

Drawn on a stratified random basis 
using established sampling para- 
meters not proportionate to the 
universe but representative of 
intended and some unintended 
audiences of NCEC products. 

* V 

, ' 1 ; 


Primary Sampling Units: 

• 8 states, randomly selected 
from major regions of 
country 

• 27 USOE-supported research 
facilities 

Secondary Sampling Units (within 
each state) : ' 

• School Districts, one large 
and one small 

• State Education Agency 

• Junior College, one per 
state 

• State 4-year College or 
University, one per state 


B. NON-RANDOM SAMPLE: 

Drawn randomly from available 
mailing or user lists of 
several dissemination channels 
for NCEC products. 

• 


• ERIC clearinghouses 

• State and local educational 
information centers 

• ERIC Document Reproduction 
service 
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• To address two evaluation objectives: assessing the "outreach* 

of the documents and evaluating their quality and utility. 

• To obtain some comparative data on the two sampling approaches 
random vs. non-random— so that recommendations could be made for 
identifying product evaluators in future evaluations. 

A Random sample could best test the outreach objective, whereas a purposive, 
non-random sample, drawn from listings that identified individuals known to be 
familiar with NCEC products, would most likely be necessary to identify product 

evaluators. 

Specialists were identified through a colleague-identification process. Nomi- 
nations were obtained from ERXG clearinghouse Advisory Board members • 

4. THE GENERAL FIELD SURVEY 

A total of 4,692 Screening Questionnaires (Qls) were mailsd, with, a return rate 
of 65 percent. Respondents were sorted into user and non— user groups on the 
basis of their answers to the question concerning prior familiarity with NCEC 
products. The number of users far exceeded that which had been originally 
anticipated: 1,837 respondents were mailed evaluation questionnaires (Q2s) . 

Potential- interest evaluation questionnaires (Q3s) were mailed to 361 non-users. 
Documents for both groups were assigned on the basis of information provided by 
respondents in their Q1 returns. Packages of 10 questionnaires, with appropri- 
ate document representations individually attached to each questionnaire, were 
mailed. The returns were 1,251, or 68 percent, for users and 255, or 71 per- 
cent, for non-users. 

5. THE SPECIALISTS* SURVEY 

A total of 242 specialists was selected from a list of nominees from ERIC 
clearinghouse Advisory Board members • Each was contacted by mail and by tele- 
phone. With some later substitutions, a total of 194 specialists participated 
and returned completed evaluation questionnaires. This return provided three 
evaluations for most of the 146 documents in the sample. 
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We believe that a clear understanding of the survey procedures is important in 
interpreting the survey findings. Detailed accounts of procedures and problems 
for each of these activities are provided in the following five sections, IIIB 

through 1 1 IF . 



I 
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THE PRODUCT SAMPLE 



An initial project task was to identify a representative sample of NCEC. infof- 
raation analysis products, including PREP reports, EMC bibliographies, and ERIC 
clearinghouse documents. Although certain assumptions could be made about 
the size and nature of the product universe, no systematically gathered and 
structured data existed for predetermining precise sampling requirements. Thus 
the sampling plan was designed to be exploratory in its approach. 



The originally planned sampling approach involved four major steps: 

• Identify approximately 500 documents included in a product 
universe limited to the period from July 1969 through 
December 1970. 

• Examine and classify documents along several dimensions, in- 
cluding product type, subject, level-of-production effort, 
and visibility. 

• Map the product universe into a two-dimensional matrix prod- 
uct type by subject — and reduce the universe approximately 
one-third for purposes of identifying a more manageable 
number of documents to analyze and characterize in-depth. 

• Perform a final reduction to approximately 75 documents, 
weighting the sample with higher level-of— effort documents 
identified by the NCEC originating sources. 



This two-step reduction process allowed adjustments in the sampling parameters 
to be made on the basis of increased insights into the shape and composition 
of the product universe. The results of the first reduction— displayed in a 
matrix of product type by subject — did, in fact, suggest the need for several 

adjustments, including the need for doubling the size of the final product 
sample and for further refining the definition of an information analysis 

product. 



> 
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The procedures used in arriving at the final product sample of 146 documents 
are discussed in four sections: 1) identification of the product universe; 

2) analysis and classification of products; 3) reductions of the product uni- 
verse; and 4) characterization of the final product sample. 



1. IDENTIFICATION OF THE PRODUCT UNIVERSE 

A universe of NCEC information analysis products was established, comprising 
hose documents prepared during an 18-month period from July 1969 through 
December 1970. This time frame was recommended by USOE, and the project staff 
concurred that these products would have been available in educational dissemi- 
nation channels for a sufficient period to be known by, and to have had some 
impact cn, the educational community. It became necessary, however, to extend 
this time frame three months for three new or reorganized ERIC clearinghouses 
(Tests, Measurement and Evaluation; Social Studies and Social Sciences; and 
Library and Information Sciences) .* 

All 18 PREP reports** and six EMC bibliographies prepared during this period 
were identified by NCEC personnel for the project staff. In identifying clear- 
inghouse documents, NCEC and the project staff used two major sources: 

• For fiscal year 1970: ERIC Products 1969-1970 , a bibliography 

of clearinghouse information analysis publications prepared by 
the ERIC Clearinghouse on Library and Information Sciences. 

• For the remainder of the 1970 calendar year: cltaringhouse 

quarterly reports to NCEC, which contain information about 
current and completed information analysis products. 



* " /"V 

It'' should be noted that this time limitation still resulted in an under- 
representation of works prepared by these clearinghouses. The universe and 
the sample were essentially one and the same, representing early efforts of 
each clearinghouse. 



For purposes of this study, the number of PREP reports in the universe was 
set at 15. One document, which was issued as four separate PREP Reports and 
as a single edition from the U. S. Government Printing Office, was created 
v as a single volume. 
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Because of several bibliographic problems involved in using these sources, such 
as multiple citations for single documents with several parts , serial publica 
lions, updated editions of documents with the same titles, and joint cl iring 
house publications, the gumber of clearinghouse documents could be only approxi- 
mated at 500. 

Copies of approximately 490 documents were assembled by h T CEC from the clearing- 
houses or, in cases where original copies were no longer available, from the 
ERIC Document Reproduction Service. This number includes clearinghouse docu- 
ments that were not listed in either of the two sources; therefore, approximately 
60 documents could not be located. The complexity of the acquisition problem 
with clearinghouse documents appeared to stem from two major problem areas . 1) the 

apparent limitations in two sources used, reflecting variations in the defini- 
tion of an information analysis product; and 2) the complexities in assigning 
"ownerstup" for documents prepared by the clearinghouse and some other agency, 
particularly their sponsoring professional associations. 

♦ 

The attempt to determine the effect of these missing documents on the complete- 
ness of the universe revealed more complexities in characterizing the product 
universe. For example, some missing documents belonged to defunct clearing- 
houses, were superseded by more recent bibliographies, or were chapters in books or 
articles in journals. However, In terms of their product types (e.g., biblio- 
graphies, reviews), the missing documents were believed to # be well represented 
in the documents on hand. Therefore, the identification and acquisition process 
was closed prior to having assembled the expected number of clearinghouse 
documents. 

These and other bibliographic control problems continued to complicate the 
development of the sample, making it difficult to be consistent in. reporting 
numbers. The creation of each successive matrix or listing revealed further 
duplications and documents listed more than once. 
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2. ANALYSIS AND CLASSIFICATION OF PRODUCTS 

The analysis and classification of documents served a twofold purpose: 1) to 

characterize the product universe in a way that would suggest a workable strati- 
fication framework for sampling; and 2) to develop a context for the meaningful 
analysis of evaluation data for each document and for groups of documents. The 
focus of this classification activity was on clearinghouse documents, which 
were greater in number and more heterogeneous in function than the PREP reports 
and EMC bibliographies. Also, only clearinghouse documents were to be involved 

in the first reduction. 

Four major dimensions or document attributes were selected for inclusion in the 
study: 1) product type; 2) subject; 3) level of effort; and 4) visibility. 

Emphasis in the first round of activity was placed on the first two dimensions 
product type and subject, both intrinsic characteristics of documents. Some 
initial efforts were made to judge the gross level of effort (relative within 
product types) and visibility of documents (from spotty and limited distribu- 
tion data provided in the clearinghouse quarterly reports). This exercise, 
particularly in level of effort,, was useful in validating product types, but 
the- actual creation of indexes in these two areas was deferred until the 
clearinghouse could provide more complete data on a more manageable number of 
documents. The two major dimfeu^ions — subject and product type — plus some 
secondary analysis considerations, are discussed in the following sections. 

The Subject Dimension . Two options were considered in selecting a classifica- 
tion structure for describing the documents by subject: 1) use of the clear- 

inghouse subject scopes (i.e., 19 general content, process, services, or level 
areas); or 2) creation of a more refined, clearinghouse-independent taxonomy. 
The decision to retain the clearinghouse subject structure was made for several 
reasons.* First, the creation of a list greater than 19 or 20 areas would 
reduce the number of possible matches with product types and therefore produce 



This decision was made only in regard to the sampling process. For purposes 
of analysis, an attempt was made to collapse the clearinghouse structure into 
several groupings (see Chapter VII) . 
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small or no numbers at each intersection in the planned matrix . This problem 
would cause serious representation problems in a final product sample of (at 
the time) only 75 products. Second, the creation of a new taxonomy would most 
likely raise problems in general concurrence, since even the present clearing- 
house structure is an area of continual discussion. 

The Product Type Dimension . The choice of product types involved a different 
set of choice points and problems, including the reconciliation of differences 
among clearinghouses in classifying their own products. An initial listing of 
ll categories was created by the project staff upon examination of some 100 
documents and after reviewing terminology used by the clearinghouses in their 
quarterly reports and in ERIC Products 1969-1970 . These product types were: 

• Bibliography, citations only . 

• Bibliography, with abstracts 

**■ * * 

• Bibliography, with annotations 

• Bibliography, with brief introductory review or analysis 

• Comprehensive bibliography, index, or abstract journal 

• Conference proceedings 

• Essays or opinion papers 

• Directories 

,* OriginaJ. research-related reports 

• Handbooks or practical guidance papers 

• Reviews and state-of-the-art papers 



The several distinctions among bibliographies were retained, on the assumption 
that they represented some variation in levels of effort. For example, the 
assumption was made that "Bibliographies with abstracts" were relatively low- 
cost documents, since the assembled materials were those prepared as a matter 
of course (and under a separate budget) for entry into Research in Edu cation 
or Current Index to Journals in Education . On the other hand, bibliographies 
with annotations, or even some citatiofc-only bibliographies, could be works of 
some original, greater effort. 
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One of the more difficult classif icati^is was the "Reviews and State-of-the-Art 



Papers." From the examination process, the staff formed strong, but necessarily 
subjective feelings about the need to distinguish among several kinds of reviews: 

1) the review that appeared to list and discuss works separately; 2)' the criti- 
cal review; 3) the state-of-the-art review in which the cited literature or 
practice was integrated into a conceptual framework for the subject or problem 
area. It was belieied, however, that examination and classification of theseif 

u 

more subtle distinctions would have to be done with the assistan.ee of subjec^ 
experts, since familiarity with the literature and knowledge base in each sub— . 

ject would be an important prerequisite. The magnitude of such a. t^sk was pro- 

t 

hibitive; thus reviews and state-of-the-art papers — long and short, critical ' 

and uncritical — are included in this product type. 

Documents were first classified by clearinghouses, and some adjustments were then 
made after documents of similar types were examined across clearinghouses. Pro- 
duct types, such as the bibliographies, were, of course, not always mutually 
exclusive. Terminology of authors, editors, or clearinghouse staff was used 
whenever possible in making final judgments. From this initial process, soine 

70 documents were eliminated from the universe. Most of these documents were 

j 

outside the time frame, were directories or ofigihial research efforts, or represented 

works in which the identification of the clearinghouse role was lost (e.g., a 

bibliography in another agency's publication; & chapter in a book; or. a multi-.. 

sponsored document in which the clearinghouse tl|«ae could not be found) . Some 

» 

peripheral documents in these groups were retained through the first reduction, 
so that the opinion o? the NCEC Advisory Board members could be solicited in 
refining the definition.* 



Other Dimensions . Two concepts wer? introduced by USOE for purposes of grouping 
documents for comparative evaluations. These two concepts were "product pairs" 



An important stimulus to the continual refinement of the information analysis 
product (s) definition was the concomitant development of the questionnaires. 
It became evident that at least two separate questionnaires should be devel- 
oped: one for bibliographies and one for substantive papers. The major 

problem, then,, was in accommodating those documents that did not draw upon 
the- literature or practice-— the basis for many of the questions to be posed. 



and ’’product sets." "Pairs” were defined as overlapping documents (i.e., over- 
lapping in subject, purpose, or intended audience) created by a clearinghouse 
and through the PREP program. The "set" "concept was aimed at looking at the 
aggregate effect of several documents created by more than one clearinghouse 
and particularly those covering some of USOE’s priority subject areas, e.g., 
the disadvantaged. For example, the'j disadvantaged-related documents might be 
drawn from any number of ERIC clearinghouses, including those on the Disadvan- 
taged, Rural Education and Small Schools, and Counseling and Personnel Services. 
Although these groupings of documents could have provided for both joint and 
differential evaluations, the "after-the-fact" construction of the groupings 
was felt to be somewhat artifical. In addition, the creation of valid group- 
ings (i.e., ones that overlapped in some specific way, or together contributed 
to some defined purpose in ajjiven priority area) required a level of partici- 
pation by subject experts and reconciliation of product type differences that, 

' ' j 

in themselves, would constitute a major undertaking. Thus, the staff attempted 
to work with these concepts through the first reductions however, with guidance 
from the Advisory Board and concurrence by USOE, they were dropped from further 
consideration 

A secondary dimension considered throughout the classification process was that 
of "intended audiences." Although some clearinghouses have more heterogeneous 
audiences than others, the stratification along clearinghouse lines provided 
some measure of control along this dimension. The exceptions to this control 
were- evident in clearinghouses that cover several different areas (e.g. , Science 
and Matbl Rural Education and Small Schools — which includes such areas as mi- 
grant education, outdoor education, American Indian education) . On request of 
a few clearinghouses, some adjustments in. the product sampling were made later 
to accommodate these diversities. 

3. REDUCTIONS OF THEJPRODUCT UNIVERSE ( 

Each of the 420 documents in the now slightly reduced universe was represented 
by a product-type and subject classification. These documents were placed in 
a two-dimensional matrix, shown in Table III— 3, to help carry out the first 

X 

III-13 

(page 14 blank) 




(C 




Bibliographies f 
Citations Only 




-1 

JL 


1 




1 








Bibliographies, 
with Abstracts 


13 

r 




1 




14 


3 


11 




Bibliographies , 
with Annotations 


~r 








2 






1 « 


Bibliographies, with 
Brief Introductory 
Review or Analysis 


2 


2 


5 




11 


'"C 

2 




i 1 

5 I 


Compr eh endive Biblio- 
graphic, Indexes or 
Abstract Journals 


1 


3 




J . . 






7 


1 1 


Essays or Opinion 
Papers 

) ' 






3 


•*" 

4 




> 

1 


10 


1 I 


f 

Original Research- 
Related Reports 




W 




# 




1 




1 I 


Handbooks or Practical 
Guidance Papers 


1 

.x> 


13 




5 


1 


4 


6 


4 1 


Reviews and State- 
of-the Art Papers 


7 


1 


9 


2 


10 


4 




3 ] 


TOTALS 


24 


20 


19 

1 


11 


39 

1 


15 


34 


20 



Q O ~< 

ERJC 



TABLE III-3. ’’UNIVERSE*' OF ERIC 




(CLEARINGHOUSE) SUBJECT AREAS 



^ 7 / / / - 












i 


1 










4 


1 


3 


1 


12 




l 

L 

r 


14 


3 


11 




1 

r 


1 


.1 


11 


6 


1 


1 I 




2 












1 


1 




2 

* I 






* 

11 


2 




5 






1 




1 




1 * 








7 


1 








2 


2 

\ 




* 1 






1 


10 


1 








2 


2 










1 




1 


2 


3 


1 




1 








1 


4 


6 


* 4 


2 


9 




2 


2 




14 I 




10 


4 




8 


17 


5 


5 


1 


7 


10 


1 i 




39 


15 


34 


20 


21 


22 


10 

/ 


22 


22 


25 


I 6 



O 

ERIC 






V 




reduction. The analysis suggested that the first reduction should be propor- 
tionate to the universe so that any weighting could be done on the basis of 
concrete data. A table of random numbers was used in selecting document codes 
(by the last two digits of an SDC-assigned code number) from each cell, with 
the objective of reducing the universe one-third. The result of this first 
reduction to 140 documents is displayed in Table I IT-4. 

At this point, the universe and first sample were compared so that requirements 
for the final reduction could be stated more precisely. With guidance and con- 
currence from USOE, the staff introduced two major shifts in the pl<^n ; 1) to 
increase the sample size to approximately 150 documents; and 2) to review the 
inclusion of several product types. Steps taken then in finalizing the product 
sample are discussed next, along with the sampling of PREP reports and EMC 

bibliographies. The exact chronology of thffe final sample development need not be 

* 

reviewed here, but the adjustments occurred over a period of time. 

Selection of Product Types . Two product types, Essays and Original Research- 
Related Studies, were immediately dropped from the sample. Each of these was 
peripheral to the definition of information analysis products and posed sig- 
nificant problems in identifying separate and distinct evaluation criteria of 
quality and utility. At a later point, the potpourri classification of expanded 
bibliographies (e.g., comprehensive indexes, abstract journals) was dropped, for 
similar reasons. Abstract journals, in particular, came into question due to 
their non-NCEC funding and differences in usage. • One final change occurred, in 
that it became necessary to place , "Bibliographies with short analyses or reviews" 
into one type or the other (i.e., a Bibliography or a Review), so that the appro- 
priate evaluation instrument — one for bibliographies and the other for substan- 
tive papers — could be used.' 

General adjustment procedures were developed to accommodate the evolutionary 
shifts in final product sizes and types and, later, to requests for adjustments 
by the clearinghouses. Because of the elapsed time, it was necessary to intro— 

t 

duce a weighted adjustment procedure prior to receiving leve 1-6 f -effort data 
from 'the clearinghouses. 
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TABLE III-*. SAMPLE OF ERIC CLEARINGHOUSE PRODUCTS FROM FIRST REDUCTION 





o 



* 



III-19 



To bring the first sample of 180 down to approximately 150, the following adjust- 
ment strategry was devised: 

1) The number of bibliographies was reduced by taking a maximum 
of two bibliographies from each of the appropriate product- 
type cells. 

2) Deleted bibliographies, essays, indexes, and original research- 
related reports were alternately replaced with Reviews and 
Practical Guidance Papers, where sufficient numbers of each 
were available. 

A table of random numbers was used to make all reductions and additions. Each 
director of a clearinghouse (or other NOEC unit) was then asked, through the 
Office of Central ERIC, to provide level-of-effort and visibility data on each 
of their documents in this latest product sample. (This data-gathering effort 
is discussed In the next major section.) From this contact, some further sam- 
ple changes were made. 

Requests from clear Inghouses were accommodated if the problem stemmed from an 

error in characterizing their universe of documents. If, on the other hand, 

% 

the request involved a significant change in sampling procedures that would 
need to be effected for all clearinghouses, changes were not made; some of 
these requests are described below: 

1) Library and Information Sciences (ERIC/CLIS) . All documents 
selected were from the previous clearinghouse contractor and 
data were no longer available. Documents prepared in the 
appropriate time frame by the new ERIC/CLIS were then selected. 

2) Disadvantaged . Only three of a series of documents considered 
by them to be a set had been sampled. The missing documents in 
the set were substituted for two other documents. 
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3) Teacher Education . The majority of reviews selected were those 
of one series. Their universe of reviews was reconstructed 
and a new sample was drawn. The problem resulted from the fact 
that one document, packaged in several -different volumes, had 
overwhelmed the original universe. 

Selection of PREP Reports and EMC Bibliographies . All PREP reports and EMC 
bibliographies were scheduled to be retained through the first reduction. In 
anticipation of the elimination of the planned secord reduction step, the 
number of PREP reports was reduced to seven, and four of the six EMC bibli- 
ographies were retained. 

The Project Officer assisted the project staff in establishing criteria for 
reduction: 1) that the PREP reports be a potential pair with a clearinghouse 

document already selected for the sample, or 2) that the subject of the PREP 
report be one of the USOE priority areas (e.g., reading, the disadvantaged, 
or the handicapped). The selections are shown in Table III-5. The table shows 
that the PREP report on Instructional TV did not meet these criteria; it was 
thus not selected for the sample. 

4. CHARACTERIZATION OF THE PRODUCT SAMPLE 

The final product sample numbered 146 documents. A complete bibliographic list- 
ing is provided in Appendix A; the composition by product type and subject is 
displayed in Table III-6. As indicated earlier, clearinghouses (and central 
NCEC units) were asked to provide level-of-preparation effort and visibility 
data on each of these documents. The data-gathering instrument used for this 
purpose is presented in Appendix B. The returned data were used to create 
level of effort and visibility indexes, for purposes of analysis, but are also 
of interest in characterizing the product sample and elements of the informa- 
tion analysis products program. 
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TABLE III-5 . UNIVERSE AND SAMPLE SELECTIONS OF PREP 
REPORTS AND EMC BIBLIOGRAPHIES 



A. PREP REPORTS 



TOPICS 


SAMPLE SELECTIONS 


"PAIRS" WITH 
CLEARINGHOUSE 
DOCUMENTS 


PRIORITY 

AREAS 


1 . 


Instructional TV 






2. 


Reading (full set) 




X 


3. 


Bilingual Education 




X 


4. 


School/ Community Relations 






5. 


Teacher Militancy 






6. 


Jobs /Disadvantaged (2) 




X 


7. 


Elementary Math 






8. 


Paraprofessionals 


X 




9. 


Sharing Educational Services 




X 


10. 


Academic Governance 






11. 


Individualized Instruction 


X 




12. 


Microteaching 






13. 


Behavior Modification 






14. 


Social Studies/Disadvantaged 




X 



B. EMC BIBLIOGRAPHIES 



TITLES 


SAMPLE SELECTIONS 


1 . 


Education Literature of the Profession (1966-1968) 




2. 


Books Related to English Language and Literature 
in Elementary and Secondary Schools 


X 


3. 


Science and Mathematics Books for Elementary 
and Secondary Schools 


X 


4. 


Books Related to Adult Basic Education and Teaching 
English to Speakers of Other Languages 


X 


5. 


Education Literature of the Profession (1969-1970) 


X 


6. 


Children's Books 1970: A List of Books for Pre- 

School through Junior High Age (joint effort 
with Library of Congress) 
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TABLE I I 1-6 . COMPOSITION OF FINAL PRODUCT SAMPLE BY PRODUCT TYPE AND (CLEAR 
INGHOUSE) SUBJECT AREA (SHEET 1) 



Explanatory Notes ; 



• EMC bibliographies 
and PREP reports 



These documents have bpen placed in the most 
appropriate (or in one. of several appropriate) 
clearinghouse subject ^reas. 



• Library and Information 
Sciences; Social Studies; 
Tests/Measurement 



- V 



The reader is ‘reminded thuL these clearinghouses 
' . ”*■!?* 

were relative!^ new Mfring the period of study . 






Exceptional Children 



Because of their 'different funding pattern and 
relation to the C^pndil on Exceptional Children, 

‘t 

they are not full? represented in, this study. 






• Educational Management 



The apparent overrepresentation of Reviews is 
due to the decision .to reclassify their 4 original] 
sampled "Bibliographies and Analyses" as Reviews. 



• Teaching of English 



The apparent overf&presentation of Bibliographies 
due to the decision to classify their originally 
sampled '’Bibliographies and Analyses" as 

l * 

Bibliographies. 



0 

ERIC 
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TABLE IT 1-6 . 



COMPOSITION 
SUBJECT (SH 



¥ 



(CLEARINGHOUSE) SUBJECT AREAS 





' * 



TABLE TIT-6. COMPOSITION OF FINAL PRODUCT SAMPLE BY PRODUCT TYPE AND 
SUBJECT (SHEET 2) 





.j 
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Level of Effort * Measures of professional labor, semi-professional/clerical 
labor, and non- labor costs were gathered from each of the NCEC units. "Profes- 
sional-man-hours" was finally selected as the unit of measurement for an index 
that distinguished among products in which the production investment was high, 
medium, and low. 

As shown in Table III— 7, professional man-hours range from 10 to 3,260 across 
all product types. The range of professional man-bours within each product 
type is shown immediately under this overview. This second display illustrates 
the fact that a "high” level of effort for one type of product could represent 
only a moderate level of effort for another type of product. As expected, 

Reviews are on the higher end of the continuum. PREP reports presented a 
slight problem in computing level of effort because there were two sources of 
cost data, from the original Targeted Communication contract and from the NCEC 
PREP report preparation. Professional effort in the original contracts was 
converted to man-hours and combined with the effort reported for NCEC personnel. 
This combined-effort rating properly placed these Reports in the high end of 
the effort distribution for Practical Guidance Papers. 

Level-of-effort indexes were created by arraying the distributions for each 
product type and establishing cutoff points that represented natural breaks or 
created reasonable distributions. For partial or missing data, the median for 
that product type was assigned to the document. The missing data reflect some 
of the difficulties in gathering such information, particularly in uniform cost 
units. In some instances, particularly for Reviews, records were available on 
the involvement of the clearinghouse professional staff but not of time spent by 
the commissioned authors. Although information on fees for commissioned authors 
was often supplied by the clearinghouses, it could not be translated by the pro- 
ject staff into professional man-hours* In other cases, data were missing for 
documents prepared by defunct organizations. 

In each cost area, the data were interpreted by the staff as best-available 
estimates. Professional man-hours were then selected because they appeared to 
contribute most to the overall document cost. Background tables on the. other 
two cost units are provided in Appendix C. The ratio of semi-prof essional/clerlcal 










III-27 



TABLE III-7. DISTRIBUTION OF DOCUMENTS IN PRODUCT SAMPLE BY LEVEL 
OF EFFORT IN PROFESSIONAL MAN-HOURS ACROSS ALL 
PRODUCT TYPES AND FOR EACH PRODUCT TYPE CLASS 



DISTRIBUTION OF 
PRODUCT TYPES 


LEVEL 


OF EFFORT 


(IN PROFESSIONAL MAN-HOURS) 


ACROSS 


10 


25 


50 


60 


100 


200 


250 


600 


10 


30 


50 


65 


102 


200 


256 


800 


10 


30 


50 


65 


112 


200 


275 


1,000 


12 


30 


50 


65 


125 


200 


300 


1,216 


12 


30 


50 


65 


125 


200 


324 


1,250 


12 


40 


50 


75 


140 


200 


325 


1,976 


14 


40 


50 


75 


145 


200 


325 


1,976 


14 


40 


50 


75 


150 


210 


339 


2,094 


14 


40 


50 


80 


150 


210 


340 


2,338 


15 


40 


50 


80 


160 


225 


345 


2,705 


16 


41 


50 


85 


161 


250 


375 


3,248 


20 


45 


50 


85 


175 


250 


400 


3,260 


20 


50 


52 


90 


180 


250 


500 




24 


50 


53 


100 


185 


250 


500 




25 


50 


60 


100 


190 


250 


570 




25 


50 


60 


100 


193 


250 


580 




25 


50 


60 


100 


200 


250 


580 




131 documents: 


Range 


= 10 to 


3,260 









Median ■ 100 



B. LEVEL OF EFFORT INDEXES BY PRODUCT TYPE 





1 

BIBLIOGRAPHIES 


PRACTICAL GUIDANCE PAPERS 


REVIEWS 


INDEX 


PROFES- 

SIONAL 

MAN-HOURS 


NUMBER OF 
DOCUMENTS 


RANGE 


NUMBER OF 
DOCUMENTS 


PROFES- 

SIONAL 

MAN-HOURS 


NUMBER OF 
DOCUMENTS 


Low: 


10-41 


22 


10-50 


10 


30-160 


25 


Medium: 


50-80 


16 


75-600 


14 


175-250 


27 


High: 


100-1,250 


9 


1,216-3,248 


7 


275-3,260 


16 




•*! 
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support to profe* 'onal ‘jtaff was fairly stable within each product type: for 

Reviews, semi-professional and clerical time was y on the average. 30 percent of 
the professional man-hours; for Practical Guidance Papers, 40 percent; and for 
Bibliographies, 60 percent. Non-labor costs (e.g., supplies, computer searches, 
typesetting) were highly variable and showed no clear relationship to other costs 
or any pattern among product types. 

Level of Visibility . NCEC units were asked to supply data on the number of copies 
distributed and to characterize the use of announcement and original distribution 
procedures for each of their documents. Indications of the types of secondary 
distribution efforts for the documents were also requested. A visibility index 
was based on the number of copies distributed (not including sales by the ERIC 
Document Reproduction Service*). PREP reports were an exception to this criterion 
of '“original distribution" as a measure of visibility. On the basis of limited 
information concerning the secondary distribution of PREP reports through the state 
education agencies, a median number of copies (65) for seven states that were 
surveyed was multiplied by 50 and added to the 250 or 300 copies distributed 
originally by USOE to the states. This gross estimate of secondary distribution 
placed the PREP reports on the high end of the continuum for Practical Guidance 
Papers. The range in number of copies distributed, across all product types, is 
displayed in Table III— 8; the individual indexes for each product type are shown 
in the same table. 



Other information provided by the states, by NCEC staff, and ERIC clearinghouses 
concerning announcement and distribution procedures were not incorporated into 
the visibility index, since there were no relevant baseline data available to 
interpret the general effect of different promotional and distribution methods. 
This information is, however, both interesting and useful for better understanding 
the information analysis product program. The methcas described are presented 
and discussed in Chapter V. 



* 

A factor for EDRS sales was not developed because records were not available for 
a sufficient period of time. The new EDRS contractor's records begin in February, 
1971. A review of the cumulative record through November 30, 1971, for 128 doc- 
uments in the product sample, showed a range of from x to 292 sales, and a median 
of 5. Since we were unable to trace a sales curve for the period since the doc- 
uments were first made available (in some cases almost 2-1/2 years ago), the 
creation of an EDRS distribution factor would have been purely speculative and 
possibly misleading. 
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TABLE III-8. DISTRIBUTION OF VISIBILITY LEVEL IN NUMBER OF COPIES 
DISTRIBUTED ACROSS ALL PRODUCT TYPES AND FOR EACH 
PRODUCT TYPE IN SAMPLE 



A. DISTRIBUTION OF LEVEL OF VISIBILITY (in number of copies distributed) ACROSS 
ALL PRODUCT TYPES 



15 


250 


428 


650 


1,071 


2,000 


3 


,550 


10,500 


50 


250 


440 


650 


1,087 


2,000 


3 


,550 


[11,000] 


100 


250 


441 


666 


1,100 


2,200 


3 


,550 


[12,250] 


100 


250 


475 


675 


1,100 


2,449 


3 


,550 


[12,250] 


130 


250 


500 


675 


1,293 


[2,500] 


3 


,550 


[12,250] 


150 


250 


500 


675 


1,300 


2,500 


3 


,550 


16,500 


179 


250 


500 


700 


1,397 


2,500 


3 


,600 


26,000 


200 


280 


500 


725 


1,400 


2,500 


4 


,000 


[35,310] 


200 


285 


500 


750 


1,403 


2,500 


5 


,000 


[35,322] 


200 


300 


500 


775 


1,450 


2,521 


5 


,000 


[36,187] 


200 


300 


551 


780 


1,627 


2,800 


5 


,000 


[40,000] 


200 


300 


600 


815 


1,810 


[3,300] 


5 


,500 


[40,000] 


200 


300 , 


600 


850 


2,000 


[3,300] 


6 


,500 


[59,728] 


200 


317 


600 


864 


2,000 


[3,300] 


7 


,500 


[60,700] 


200 


350 


600 


875 


2,000 


[3,300] 


7 


,626 


[62,500] 


200 


350 


610 


95U 


2,000 


[3,300] 


8 


,477 


[63,133] 


209 


400 


630 


950 


2,000 


[3,300] 


9 


,712 


250 


400 


650 


1,000 


2,000 


3,500 


9 


,920 





[Brackets indicate inclusion of journal circulation or subscription 
series figures] 



For 142 documents: Range = 15-63,133 

Median * 950 



B. LEVEL OF VISIBILITY INDEXES BY PRODUCT TYPE 



INDEX 


BIBLIOGRi 


\PHIES 


PRACTICAL GUIDANCE PAPERS 


REVIEWS 




NUMBER OF 
DOCUMENTS 








r ■*— — 




COPIES 


RANGE 


NUMBER OF 
DOCUMENTS 


COPIES 


NUMBER OF 
DOCUMENTS 


I- w: 


100-950 


25 


15-350 


10 


50-950 


29 


Medium: 


1 2,000-10,500 


15 


400-1,627 


10 


1,000-2,521 


25 


High: 


35.310-63,133 


7 


2,000-11,000 


11 


3,300-40,000 


14 
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THE SURVEY INSTRUMENTS 



C . 

Four different survey instruments were designed and pretested for use in the 
two surveys (see Table III-l) . Complete questionnaire packages are contained 
in Appendix D of this report, and each is described below. 

1. GENERAL FIELD SURVEY QUESTIONNAIRES 

The General Field Survey required the development of a Screening Questionnaire 
and a User Evaluation Questionnaire. The Screening Questionnaire (Ql) in 
which the survey participants were asked to indicate their general familiarity 
with NCEC products, was intended to help identify potential evaluators of 
these products. This instrument was originally planned as an "unprompted re- 
call" questionnaire, i.e., one in which specific product titles or descrip- 
tions would not be presented. However, it was later decided that some stimulus 
— in this case, an insert section of miniature color replicas representing 
examples of products — would be added to help the survey participant in 
distinguishing among the many kinds of information products available. It was 
hoped that this approach would stimulate greater and more accurate recognition. 

The User Evaluation Questionnaire took on several forms. For users, i.e., 
those who indicated in Ql that they had read or skimmed NCEC products, two 
questionnaire versions (of Q2) were designed: Q2-A, for substantive papers 

— Reviews and Practical Guidance Papers — and Q2-B, for Bibliographies. For 

non-user respondents to Ql, i.e., those who indicated they had only heard 
about or had not previously read any NCEC products, an abbreviated form (Q3) 
was designed to explore their potential interest in the documents. 



A more detailed discussion of the purpose and design of each of these three 
questionnaires is contained in the following sections. 

Screening Survey Questionnaire (Ql) . The Screening Questionnaire was a brief 
four-item instrument that served two major purposes: 1) to identify a sub- 

sample of educators who had read NCEC products, and 2) to obtain a minimal 
amount of demographic data that would describe the respondent population and 




* 



* * 
w/ 
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provide information for assigning documents in the evaluation phase, 
questions asked were: 



The four 



1) In what institutional setting are you working? 

2) What is your main professional role or function in the 
educational community? 

3) Which of the following [34] areas represent your major 
professional interests? [Circle the one area that 
represents your principal professional interest] 

4) Prior to receiving the questionnaire, had you seen or 
read an information product prepared by any one of these 
[21] NCEC units? 

The categories of broad interest areas in the third question were selected to 
facilitate the assignment of documents. Clearinghouse names were used for 
directly matching relevant documents. Separate categories were created in 
cases where a clearinghouse scope was not adequately encompassed by the 
clearinghouse name (such as the Rural Education and Small Schools Clearing- 
house’s coverage of American Indian Education) or where two different areas 
were included in one clearinghouse name (such as Science and Mathematics 
Education). Other key words were added to assist in focusing on the appro- 
priate level (elementary education or secondary education) or on the type of 
document (e.g«, related to research, curriculum development, or instructional 
materials). Several areas not covered by the clearinghouses were also added 
in anticipation of some responses from unintended audiences. 

The fourth question was the key screening item. Respondents were given three 
response choices for each of the 21 NCEC units (all 19 ERIC clearinghouses, 
PREP reports, and EMC bibliographies); 

• I have read or skimmed 

• I am aware of, but have not read or skimmed 

• I am not aware of 
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Respondents who checked the first response for any one NCEC unit were considered 
users; those with responses in only the second or third choices were classified 
as non-users. Users were potential product evaluators for the study and would 
receive evaluation questionnaires; non-users, on the other hand, were to be 
followed up in hopes of learning of their potential interest in the documents 
and, in cases where they had heard about the documents, why they had not 
read them. 

In the last question, several recall prompters were supplied: background 

information concerning the preparation and original dissemination channels 
was provided; acronyms of NCEC units were given with locations and/or institu- 
tional settings of ERIC clearinghouses; and, as mentioned before, miniature 
color photos of the products were provided in a special insert section. 

User Evaluation Survey Questionnaire (Q2) . Evaluation questionnaires were 
developed for use with specific document representations (i«e«, one-page 
displays of title pages, tables of contents » and abstracts or extracts) of 
documents for a ''prompted recall 11 evaluation by respondents. A sample document 
representation is presented in Exhibit III— 1; copies of several are included 
in Appendix D with the questionnaires. 

Two versions of Q2 were developed in order to reflect different evaluation 
criteria for the different purposes and uses of Bibliographies and substantive 
papers (i.e., Reviews and Practical Guidance Papers). Although the overall 
dimensions of quality, utility, and impact ware explored through both of these 
instruments, only six questions were phrased alike in both. An overview of 
questionnaire items used in all evaluation instruments is displayed in Table 
III-9. (The Potential-Interest Evaluation Questionnaire and the Specialists' 
Questionnaires [Q3 and Q4] discussed next, are also included in the table.) 

In the evaluation survey instruments, questions for each of these dimensions 
were carefully phrased in terms of the user's own needs, at the time of his use. 
Although this approach exacted a burden on the reader in recalling his "in- 
formation-need environment," we felt that this approach more nearly 
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EXHIBIT III-l. 



SAMPLE DOCUMENT REPRESENTATION USED IN USER EVALUATION SURVEY 
QUESTIONNAIRE (Q2) 



Social 

and 



Edited by 



[KA 039 ] 



Philip K. Pielc Terry L. Eidcli 

Director, ERIC/CEA Research Associate , CASEA 



Technological 

Change 

Implications for Education 

{drawn from Title Page] 



with 



'Stuart C. Smith 

'! ‘ s * , \ 

Assistant Director and Editor, ERIC ICE A 

Tnf CcwTn for the Advanced Sun 
. *f Educational Administration 
W rtVERsnr or Oregon, Eugene, Oregon 

1970 



» infclly, each author waa expected to project the future development of 
knowledge in hie topic, baaed on hie reatiatic a«*e**ment of current knowl- 
edge, iu hiatorical development, and the probable aorial context of the [extract from 

future. Projections were not expected to exceed • period of more thin on# Edi tors Introduction J 

decade. 

{partial display of sub-headings] 



fart u Nature of Our Changing Socncrr: 
Implications for Schools 
JT. Harman 
1. Introduction 

1 Apparent Long-Term Trends 
3. Two Contrasting Forecasts 



In ptrt l, Willie W. Harmsn present* t broad overview of our chang- 
ing society «nd it* implications for the future of education. Harman 
disavow* any attempt to predict the future, and inttead seeks to provide 
a conceptual framework for understanding the direction and nature of 
plausible "alternative future**’ for society. 



part n. Teacher Militancy: 



1 

*4 



Implications for the Schools 



Richard C . Williams 






9. Introduction 

10. Level of Teacher Militancy 

11. Conditions Affecting Teacher Militancy 



If Harman can b# said to take a macroview of soda! change, a micro* 
▼few is provided by Richard C Williams in part 2. Williams begin* with 
an analysis of internal and external condition* contributing to the rite 
of teacher militancy in the public schools. 

Following this analysis, Williams describes thres altemativs models 
for improving the involvement of teacher* in the decisionmaking pro£$s. 



fart ni. System Approaches to Education: 

Discussion and Attempted Integration 



Roger A . Kaufman 

15. Introduction 

16. Why a Sy*tem Approach? 

17. Education and Management: Design -Process Mod# 



u 



V 

m 

u 



In part 3, beginning the technologic action of the monograph, Roger 
A. Kaufman seekj to promote a comma*, understanding among adminis- 
trator* as to the use* of the system approach in solving educational prob- 
lem*. Kaufman define* a *ystem approach a* the application of formal 



fart iv. System Approaches to Educational Planning 
Marvin C. A Ik in and Jame$ £. Bruno 



23. Introduction 

24. Operation* Research 

25. Planning -Programming-Budgeting Systems 



fart v. Educational Management Information 
Systems: Procressand Prospective* 
John A, Evans 

30. Introduction 

31. Definition of Concepts 

32. Developments in Management Informed or 
S ystem Technology 



Marvin C. Alkin and James E. Bruno, in part 4, discus* application* 
of systems approaches to educational planning. Alkin and Bruno focua 
oq that aspect of planning primarily concerned with internal decision* 
in education, Le., decision* that involve making choices among alterna- 
tives, methods, media, and toebnot o g***. 



Since a systems study is only as good as the date it employs, part 5 of 
this collection, by John A. Evans, on educational management informa- 
tion systems, is an appropriate conclusion to the technological *ection of 
r . the monograph. 

Intending to contribute & better understanding of computer-based 
management information systems (MIS) and their implication* for edu- 
-rational management, Evan* define* and clariCaa major terms and con- 
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corresponded to the objective of assessing the general utility and impact of 
the documents, and accommodated the inevitable variations in users’ levels of 
knowledge in the field or familiarity with literature. An in-depth review of 
quality was expected from participants in the Specialists' Survey. 

Each respondent was asked to review 10 documents, but it was anticipated that 
no more than two or three documents would have been read. If a document had 
not been read previously, the respondent was asked to answer only two or three 
question^. The first two covered the potential relevance and potential useful- 
ness of the document at the time of publication; the third asked why a respon- 
dent had not read a document that he had heard about. 

The items on the Potential-Interest Evaluation Questionnaire (Q3) were compa- 
rable to those for non-readers in Q2. Non-users were also asked if they had pre- 
viously read the documents. This question would serve to Lest the reliability 
of the key screening item on Ql. 

2. SPECIALISTS' SURVEY QUESTIONNAIRE 

Specialists were to receive full copies of documents and provide in-depth 
assessments of their quality and general utility. The Specialists’ framework 
for responding was not to be limited by personal information needs; it was, 
rather, to encompass the needs of the field relative to the extant literature 
and practice. As shown in Table III-9, some nine questions were generally or 
directly comparable to those asked of the General Field Survey participants. 

Two questions were designed to allow the evaluator to disqualify himself because 
of specific involvement in the preparation of the document (e.g., authored, 
edited, or reviewed prior to publication) and/or because of the document’s lack 
of relevance to his professional interests. As background information, an item 
was included to determine whether the evaluator had read t)he document previ- 
ously and, if so, how recently. 
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The assessment of ten different quality dimensions listed under Quality in 

Table 111-9 was elicited through a combination of ratings and open-ended 
sections. The free-form responses allowed the specialist to cite specific 
examples from the text in support of these ratings. In a context of recom- 
mending the document to a colleague, evaluators were also asked to rate a 
document's degree of usefulness to the educational community for seven specific 
dimensions of utility. As in the User Evaluation Questionnaire (Q2) , a 
concluding question asked for a general comparison of the document's usefulness 
in relation to other documents of the same type. 
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D. 



THE SURVEY PARTICIPANTS 



As discussed in the introduction to this chapter, two separate surveys were 
conceived in order to explore fully the questions of quality, utility, and 
impact of NCEC products. Questions of outreach into the educational community, 
of general usage and quality, and of impact on the daily activities of educators, 
could best be posed to a broadly based sample of educators. On the other hand, 
specific questions of quality, relative to the literature and knowledge base as 
a whole, were more appropriately addressed by a segment of the educational 
community specifically identified as "specialists." The latter type of 
respondent would certainly constitute some portion of any general sample; 
however, it was important to identify these individuals as a separate gfoup 
and to request a somewhat different, more in-depth evaluation of the products. 

In addition to such considerations that led to the two-survey approach, there 
were equally compelling reasons for generating a dual sampling approach in 
identifying participants for the General Field Survey. It was believed that 
a random approach would provide data on answering the "outreach" question, but 
that a non-random sample — one that included individuals known to be familiar 
with the educational information channels — would most likely produce potential 
product evaluators. This dual approach was also viewed as a means for deriving 
some comparative data that could help in determining the more efficient and 
cost-effective method of identifying product evaluators in future evaluations. 

The following sections trace steps used in identifying survey participants for 
both surveys: 1) The General Field Survey, and 2) The Specialists’ Survey. 

1. THE GENERAL FIELD SURVEY 

General Field Survey participants were identified through two sampling approaches 
1) a random sample drawn on a stratified basis from several primary and sec- 
ondary sampling units, and 2) a non-random sample drawn from listings avail- 
able within the "NCEC family." Since there is no one central list of educators, 
the following sampling procedures reflect a multi-stage and multi-unit sampling 
approach. 
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Development of the Random Sample, A random sample of 1,588 educators was drawn 
according to the general parameters shown in Table III-1Q. 

Table III-10. GENERAL PARAMETERS FOR THE RANDOM SAMPLE OF THE . 

GENERAL FIELD SURVEY 



FUNCTION OR ROLE 
IN THE EDUCATIONAL 
COMMUNITY 




SAMPLING 


UNITS 






School 

Districts 


State 

Education 

Agencies 


Institutions 
of Higher 
Education 


Research 

Facilities 


TOTAL 

PERCENTAGE 


Teaching Personnel 


30% 


- 


5% 


- 


35% 


Support Personnel 


15% 


5% 


5% 


- 


25% 


Administrative Personnel 


15% 


10% 


•5% 


- 


30% 


Researchers 


- 


- 


- 


10% 


10% 


Totals 


60% 


15% 


15% 




10% 


100% 



This sample was not intended to be proportionate to the universe of educators, 
since the largest percentages, i.e., elementary and secondary teachers, are 
relatively homogeneous strata and represent only segments of the intended 

audiences of the NCEC products. Administrators and support personnel were 

sampled at higher percentages than their strength in the universe to provide a 

representation of significant audiences and important links in the information- 
utilization chain. 



o 

ERiC 
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Two primary sampling units were used: 

• 8 states were randomly selected from each of the major 
regions of the country. 

• 27 USOE-supported research facilities were used as the 
primary units for identifying research and development 
staff members. 



Four secondary sampling units were used within each of the eight states: 

1) the State Education Agency, 2) two school systems — one large (over 10,000) 
and one small (under 10,000), 3) one junior or community college, and 4) one 
public four-year college or university.* * The specific sampling parameters and 
numbers actually sampled for each unit are fully documented in Appendix E. 



Listings of personnel for each of the sampled units were obtained through 
mail and telephone contacts with agency personnel. The contact process with 
personnel in each appropriate agency is summarized in the following paragraphs: 



• State Education Agencies . In each of the state agencies, the 
PREP contact (an individual identified by the Chief State 
School Officer as the State’s liaison with NCEC) was telephoned 
initially. It was not only important that these key people be 
made aware of the study, but it was felt that their support 
could be most helpful, particularly if there were problems in 
enlisting local support. These individuals were most cooperative 
in providing state directories or personnel listings. One PREP 
contact volunteered to initiate the request with the local school 
districts in that state. No formalized requests were required 
for surveying the state personnel. 



Each of the last three sampling units was sampled from the most current 
editions of directories prepared by the U. S. Office of Education's National 
Center for Educational Statistics: Education Directory, Public School Systems 

1970-1971 and Directory of Higher Education . 
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EXHIBIT III-2 . LETTER FROM USOE PROJECT OFFICER TO SCHOOL DISTRICT OFFICIALS 

REQUESTING COOPERATION IN SURVEY 




DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE 
OFFICE OF EDUCATION 
WASHINGTON, b C £0202 



Dear Educator: 

The L'.S. Office of Education has contracted with System Development 
Corporation (SDC) to conduct an evaluation study of information 
analysis products prepared by our National Center for Educational 
Communication. These products include bibliographies, state-of- 
the-art papers, and handbooks prepared by ERIC (Educational 
Resources Information Center), the PREP (Putting Research Into 
Educational Practice) Reports, and bibliographies from the 
Educational Materials Center. The purpose of this study is to 
have educators throughout the country assess the quality and 
utility of these products, so that USOE can determine how future 
products can be targeted to meet the needs of the professional 
community. 

Your school district has been chosen as one of 16 districts in the 
country from which SDC will select professional educators to 
participate in this study. The enclosed letter from SDC explains 
further the type of survey in which your personnel would be 
involved. We at USOE would very much appreciate your local 
support of this study and your cooperation in making available 
your personnel lists to SDC. 

USOE is confident that this study will not only help us determine 
the impact we have had in the field thus far, but most importantly, 
help us to improve our efforts in the near future. Thank you for 
your cooperation. 

Sincerely, 



(Mrs.) Betty Rasmussen 
Office of Program Planning 
and Evaluation 
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Development of the Non— Random Sample . A non— random sample of 3,221 was drawn 
from several sampling units, each of which represented a dissemination channel 
for NCEC products. The three basic groups were: 

• The ERIC Clearinghouses 

• State and Local Educational Information Centers receiving 
support from USOE 

• The ERIC Document Reproduction Service 

Procedures for obtaining these various listings, and the complexities encount- 
ered in the process, varied in each of these basic groups, as described in the 

following sections. 



• ERIC Clearinghouses . Each clearinghouse was initially contacted 
by telephone for purposes of obtaining a "mailing list." From 
these initial discussions, the staff was made aware of the dif- 
ferences in listings for individual clearinghouses and the nature 
of what was, for some, a burden. The differences were: 1) in 
size, with lists ranging up to 27,000 names, some few of which 
were stratified; 2) in formats* with most lists in machine- 
readable form, but at least one existing (at the time) on typed 
3x5 cards; and 3) in sources of ownership, for at least seven 
clearinghouses where professional assocations or mailing services 
maintained their listings. Thus, several refinements tc the 
request were introduced. The mailing list was defined 
as the broadest listing of intended audiences available, to 
exclude any limited or automatic distribution lists (e.g., to 
USOE, other clearinghouses, or Advisory Board personnel). Also, 
to facilitate the delivery process — particularly for those 
clearinghouses with large computerized listings— they were 
asked to sample 100 or 200 names randomly.* If more convenient, 
a complete listing was provided for sampling by the project staff. 

This request produced approximately 2,436 names (after foreign 
addresses and institutional names were eliminated). 



Twelve clearinghouses that appeared to have more heterogeneous intended 
audiences were asked to sample 200 names, so that this diversity could be 
represented in the sample. 
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During the course of these several contacts, some clearinghouses 
suggested additional types of "lists" that would facilitate the 
identification of specific product users, those in which indiv- 
iduals who receive copies from the clearinghouses were identified. 
However, not all clearinghouses maintained this expanded auto- 
matic distribution list, and for some, the identification of 
known product recipients would require a manual search of request 
files. Thus, on a voluntary basis, the clearinghouses were asked 
to supply approximately 10 known product users for each document, 
if the request could be filled within a reasonable level of 
effort. Only eleven clearinghouses supplied 284 names through 
this second request; thus, all documents in the product sample 
were not covered in this listing. 

• State and Local Educational Information Centers . Fourteen state 
and local educational information centers were contacted initially 
for assistance in providing lists. Again, the problem of defining 
was encountered. Most of these centers do not maintain 
announcement or mailing lists, since they operate more directly 
with on-demand services. Therefore, it was more appropriate for 
most of the local centers to provide names of 50 of their most 
recent (not necessarily most frequent) users. For some of the 
state centers, the request was apparently not reasonable, par- 
ticularly for those which work through satellite centers or 
dissemination agents. Four centers contributed a total of 362 
names; the remaining centers either did not choose to cooperate 
or did not have an appropriate listing. 

* ERIC Document Reproduction Service (EDRS), . EDRS was initially 
discounted as a possible source on the assumption that most 
customers would represent information intermediaries or purchas- 
ing agents for institutions. However, the potential of this 
source was explored further, and records provided by LEASCO 
Information Products (the EDRS contractor) were used as a sampling 
source. First, from a printout of ED numbers (ERIC accession 
numbers), 696 customer order numbers were sampled. From a seen 
printout, these numbers and customer order numbers were translate 
into names of purchasers. At this second stage, 390 names were 
lost to duplications or foreign addresses. Of the remaining 306 
purchasers, 55 percent were institutional addresses. The remain- 
ining 139 individuals, including approximately nine percent 
librarians, were included in the survey mailing. 

The total sampling picture for both sample groups is presented 
in Table III-ll.* 



*These numbers differ slightly from those displayed in a later table (see section 
S) of actual mailings because duplicate names were eliminated prior to the 
actual mailings. 




O 
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TABLE III- 11. NUMBER OF GENERAL FIELD SURVEY PARTICIPANTS IDENTIFIED BY 

SAMPLING UNITS WITHIN RANDOM AND NON-RANDOM SAMPLES 



Sampling Units 


Number of 
Sampling Units 


Total Number of 
Individuals Sampled 


RANDOM SAMPLE 






State Education Agencies 


8 


233 


Small School Districts 


8 


292 


Large School Districts 


8 


624 


Junior Colleges 


8 


102 


State Universities 


8 


143 


USOE-Supported Research Facilities 


27 


194 


RANDOM SAMPLE TOTALS 

0 




1588 


NON-RANDOM SAMPLE 






ERIC Clearinghouses 

(General Mailing Lists) 


17 


2436 


Local Educational Information Centers 


4 


199 


State Educational Information Centers 


4 


163 


Requesters of Documents in Product 

Sample from ERIC Clearinghouses 


11 


284 


Customers of ERIC Document 
Reproduction Service 


1 


139 


NON-RANDOM SAMPLE TOTALS 




3221 


TOTALS FOR BOTH SAMPLES 




4809 
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2. THE SPECIALISTS' SURVEY 

Specialists were identified through a structured colleague- identif ication process. 
Several possible starting points for this process were considered, e.g., pro- 
fessional association officers and masthead listings of editorial advisors for 
professional journals. However, in keeping with the methodological considera- 
tions for future evaluations, a starting point "close to home" was selected: 

ERIC Clearinghouse Advisory Boards. Each clearinghouse provided lists of 
Board members- Letters were then sent to a total of 248 members requesting 
that they identify individuals who, in their opinion, would qualify as subject 
specialists in the areas covered by the documents.* The letter of request was 
accompanied by a listing of documents for the appropriate clearinghouse and, as 
shown in Exhibit III-3, a recommendation form was enclosed so that individuals 
could provide names, addresses, and telephone numbers of nominees. In addition, 
Board members could recommend individuals as "generalists," or as particularly 
qualified evaluators of specific documents. A project description was also 
enclosed to assist members in better understanding the purpose and methodology 

of the study. 



From approx im ately 100 Advisory Board members, a total of 482 nominations (in- 
cluding some duplications) were rec.l/ed. The goal was to identify three 
specialists for each document, so that a "tie-breaker" would be available for 
any two diametrically opposed evaluations. Thus, for each clearinghouse area, 
specialists were taken in the following order: 

• Those nominated three or more times 

• Those nominated at least twice and recommended for one 
or more specific documents 

• Those nominated twice 



*Not all Advisory and Policy Board members of clearinghouses represent subject 
expertise in their advisory capacity; because of this, some members disqual- 
ified themselves from participating in this recommendation process. 



* 












EXHIBIT III-3. 



LETTER OF REQUEST TO CLEARINGHOUSE ADVISOR BOARE> MEMBERS 
FOR NOMINATION OF SPECIALISTS. (SHEET ' 








SYSTEM DEVELOPMENT CORPORATION 

582? Columbia Pike, Falk Church. V»rgmta 22041 






-r 



ember 6, 1971 



Dear ERIC Clearinghouse Advisory Board Members: 

System Development Corporation, under contract to the U.S* Office of 
Education, is conducting an evaluation of selected information analysis 
products prepared by the National Center for Educational Communication 
in USOE. The sample of 150 documents*" selected for evaluation Includes; 



• ERIC products (e.g, 
the-art papers) 



bibliographies , state-of - 



• PREP (Putting Research Into Educational Practice) 

Reports 

* a; 

« A 

• Educational Materials Center bibliographies 

* 

Our study involves a General Field Survey of 5000 educators and a 
Specialists’ Survey. It is in regard to the Specialists' Survey that 
we are seeking your assistance. - * , 

■ ; . ^ 

In the attached Project Description, several aspects oiythe study otjjec- 
tives and design are discussed. Specif ically^we need. help in ' 
identifying subject specialists to serve as paid Jrvaluitdrs . To 
complement the utility-related evaluations jre Itojj^btaio from the 

General Field Survey, we will ask selected ''speci to work with the 

full text of the .documents in their subject areas* tgt provide us with 
assessments of the documents' quality. Our goal is to obtain evaluations 
from three different specialists for each document. To obtain the names 
of individuals who qualify as subject specialists ,. wp have chosen to use 
a colleague-identification process. We believw ^bat as educational 

leaders with special knowledge of the ERIC system, are! tiSe key people 
with whom we should initiate this. identification process. 

We have purposely refrained from defining - * specialty in terms of 
specific criteria of institutional settings,' years in the field, 
publications authored, etc. We feel j that each- of you will define 
’specialist* in a way that combines these type, of external criteria 
with your own judgments of an individual's professional qualifications. 

We have also not developed any exclusion criteria. Advisory Board 
members and document authors are certainly prime candidates for nomination. 



EXHIBIT III-3 . LETTER OF REQUEST TO CLEARINGHOUSE ADVISORY BOARD MEMBERS FOR 

NOMINATION OF SPECIALISTS. (SHEET 2) 



(In the case of document authors we would, of course, be careful not 
to assign their own publications to them!) Please feel free to nominate 
yourself, for we want to Identify specialists who are interested in the 
project and who will take the time to provide us with thoughtful and 
careful assessments. 

To help you in this nomination process, we have enclosed a list of 
documents included in the product sample for the clearinghouse with 
which you are affiliated. Documents on these lists are identified by 
title, author, publication date, descriptors from Research in Education . 
ED number, and product type (e.g., bibliography, state-of-the-art paper). 
Attached to the list is a Recommendation Form for you to use in providing 
us with the names, addresses, and telephone numbers of nominees. We 
have also provided a column for you to match individuals with one or 
more specific documents for which you feel specialized expertise is 
required. If this column is not marked, we will assume that the nominee 
is a generalist who could handle most of the other documents listed. 

For your convenience, we have enclosed a prepaid return envelope. 

After we receive your nominations, we will personally contact these 
individuals to invite their participation in the study and to discuss 
such matters as remuneration, the evaluation instrument, and scheduling. 
We will keep all responses to our evaluation questionnaire in strictest 
confidence; no responses will be attributed to a particular individual. 

Should you have any questions, please feel free to call Miss Mary Jane 
Ruhl or me collect at (703) 820-2220. Thank you for your assistance. 




Sincerely, 

. "7 



JW/lsr 



Enclosures 



Project Director 

NCEC Product Evaluation Project 



EXHIBIT III-3. LETTER OF REQUEST TO CLEARINGHOUSE ADVISORY BOARD MEMBERS FOR 
NOMINATION OF SPECIALISTS. (SHEET 3) _____ 

RECOMMENDATION FORM 

NCEC Product Evaluation Specialists' Survey 

Name Address Telephone Recommended onlv for 

Specific Document(s) 
(Please identify by 
document number) 

1 . 



3 . 



4 . 



5 . 



□ Check here if we may use your name in contacting these individuals. 



Name 



Clearinghouse Affiliation 



0 

ERIC 
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[If needed , please use reverse side.] 



• Those recommended for one or more specific documents 

• Those recommended only once, and not for any specific 
documents (individuals in this group were randomly 
sampled to complete the necessary number of specialists 
and for that clearinghouse’s documents, including subject 
related PREP and EMC documents). 



A total of 242 specialists, representing three potential evaluators for all 146 
documents, were identified. The contacting process and refinement of the docu- 
ment assignments are discussed in Section F of this chapter. 
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E. THE GENERAL FIELD SURVEY 

This section describes the activities related to the conduct of mailings and 
the handling of returns for each phase of the General Field Survey. The 
results of the survey are summarized in Tables III-12, III-14, and 111-15. 

1. SCREENING QUESTIONNAIRE SURVEY — PHASE 1 

Mailings . A total of 4,692 Screening Questionnaires (Qls) were mailed. A 
master control file was checked for duplicates within and between samples, 
and between the General Field Survey and the Specialists Survey.* A total of 
117 duplicates was identified and pulled. The initial mailing was accom- 
plished in three separate waves, because of delays in obtaining groups of 
source listings for sampling. Alphanumeric codes were assigned for the 
identification of each sampling source and each individual. The entire Q1 
package consisted of questionnaire (with cover letter), a color insert of 
sample products, and a pre-paid envelope. Followup packages contained the 
same materials with a special cover letter. 

Returns . As shown in Table III-12, the overall response rate, with one 
followup mailing, was 64%. This table shows only usable returns; during 
the check-in process, some 74 returns were discarded for one of several 
reasons. Typically, these questionnaires were either incomplete, not com- 
pleted at all (for such reasons as "I am not an educator" or "I do not 
complete anything without being paid") , or completed by someone other than 
the originally designated survey participant (e.g., some directors of 
administrative units seemed to have passed the questionnaire to a sub- 
ordinate or librarian, an associate, or a successor). 

Editing and Keypunching . Guidelines for editing each questionnaire were developed 
to help achieve uniformity in preparing data for analysis. These were developed 
on the basis of an initial review of some 100 questionnaires. Although multiple 



ft 

A priority of placements was used co control the removal of duplicates; in 
order, these were: (1) Specialists; (2) Random sample names; (3) Non-random 

sample names. 
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TABLE III-12 



SCREENING QUESTIONNAIRE (Ql) MAILINGS AND RETURNS BY SAMPLING 
UNITS 





Nuaber 




Number 


Sampling Units 


Mailed 


Returned 


RANDOM SAMPLE 






State Agencies end Institutions 






Connecticut 


177 


103 (56%) 


Florida 


180 


94 (52%) 


Indians 


162 


101 (62%) 


Kansas 


173 


106 (61%) 


Maryland w 


179 


99 (55%) 


Texas 


174 


95 (55%) 


Utah 


176 


105 (60%) 


Washington 


173 


95 (55%) 


USOE-Supported Research Facilities 


190 


136 (72%) 


RANDOM SAMPLE TOTALS 


1584 


934 (59%) 


NON-RANDOM SAMPLE 






ERIC Clearinghouses 






Adult Education CH 


88 


53 (60%) 


Disadvantaged CH 


175 


83 (47%) 


Early Childhood CH 


167 


100 (60%) 


Educ. Managesent CH 


192 


115 (60%) 


Media/Technology CH 


154 


110 (71%) 


Higher Education CH 


88 


64 (73%) 


Junior Colleges CH 


121 


95 (79%) 


Languages /Linguist ica CH 


118 


85 (72%) 


Library/Info. Sciences CH 


136 


80 (59%) 


Reading CH 


76 


55 (72%) 


Rural/Small Schools CH 


180 


142 (79%) 


Science/Mathematics CH 


185 


131 (71%) 


Social Science CH 


93 


55 (59%) 


Teacher Education CH 


195 


127 (65%) 


English CH 


98 


60 (61%) 


Tests /Measurement CH 


100 


69 (69%) 


Vocational/Technical CH 


193 


114 (59%) 


Local Educational Information Centers 






Bay Area Information Center 


50 


33 (66%) 


(Redwood City, California) 






Information Retrieval Center 


42 


32 (76%) 


(Boulder, Colorado) 






Merrimack Educational Center 


48 


32 (67%) 


(Chelmsford, Massachusetts) 






RISE Center 


50 


37 (74%) 


(Conahohocken, Pennsylvania) 






State Educational Information Centers 






Florida 


30 


23 (77%) 


Oregon 


48 


40 (83%) 


Texas 


29 


20 (69%) 


Utah 


52 


41 (79%) 


Requesters of Documents (in Product 


265 


187 (71%) 


Sample) from ERIC Clearinghouses 






Purchasers of Documents (in Product 


135 


96 (71%) 


Ssmpie) fro© ERIC Document 






Reproduction Service 






NON RANDOM SAMPLE TOTALS 


3108 


2079 (67%) 


TOTALS FOR BOTH SAMPLES 


4692 


3013 (64%) 
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responses had been invited in the first two questions — concerning setting and 
role/function — the decision was made to simplify the characterization of the 
respondent population by matching one setting with the single principal role/ 
function. In cases where the respondent’s principle role was not indicated 
by the checkmarks, the staff selected that role which appeared to have the 
highest "information stress." For example, in a "researcher"/"professor" 
combination, the "researcher" would be selected. In the case of the 
"principal"/"supervisor of instruction," the "principal" would be selected. 

All responses were accepted for the third question, in which a respondent 
was asked to check each of his major interest areas and to circle the one 
representing his principal interest. If more than one interest area was 
circled, or none, selections were not made oy the staff; for these respondents, 
no one major area is represented in the file. 

The fourth question, the key familiarity question, required no special editing. 
If a respondent checked more than one column for a particular unit, the 
assumption was made that the first check was more accurate. 

Letters of appreciation for cooperation were sent to all Q1 respondents from 
the USOE Project Officer. These letters also served as an "alert" for the 
second-phase survey, discussed next. The letter to users is shown in 
Exhibit III-4. 

2. EVALUATION QUESTIONNAIRE SURVEY— PHASE 2 

Selection _oi Product Evaluators . In preparation for mailing the User 
Evaluation questionnaires (Q2 and Q3>, responses to the Screening Questionnaire 
were checked in and sorted as users and non-users. Users had indicated reading/ 
skimming products from at least one NCEC unit; non-users had not. (The 
question of general familiarity among respondents is addressed in Chapter V.) 

At the outset of the study, it was assumed that approximately one out of 
four respondents to Q1 would have read/skimmed products. The high number of 
user returns prior to any followup — approximately 1,800 — was surprising and 
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EXHIBIT III-4. 



FOLLOWUP LETTER FROM USGE PROJECT OFFICER TO USER RESPONDENTS 
OF Q1 




DEPARTMENT OF HEALTH, EDUCATION. AND WELFARE 

OFFICE OF EDUCATION 
WASHINGTON, D C 20202 



February 1, 1972 



Dear Educator: 

You may recall from the questionnaire you so kindly completed 
several weeks ego, that a major national effort is being conducted 
by System Development Corporation (SDC) , under sponsorship of the 
U.S. Office of Education (USOE) . The purpose of this study is to 
evaluate the information products of US0E f s National Center for 
Educational Communication (NCEC) . On behalf of SDC and our Office, 

I would like to thank you for your response and explain briefly the 
next phase of the project. 

The questionnaire which you completed was sent to approximately 5,000 
educators throughout the country. It is being used to assess the 
general level of awareness of NCEC products, such as PREP reports, 
Educational Materials Center bibliographies, and ERIC documents. SDC 
is now inviting respondents who indicated some familiarity with NCEC 
products to assist in the second ph?se of the study. In the next 
week or so, SDC will send you a second mailing in which you will be 
asked to answer several questions about 10 NCEC products. 

To help you in recalling those with which you have some familiarity, SDC 
is including ’'representations” of the documents you are asked to 
evaluate. After a quick examination of these representations, you are 
asked to supply answers, mostly check-mark responses, to the attached 
questionnaires. You are not expected to have read all 10 documents. 
SDC’s pre-tegt participants spent an average of 45 minutes to complete 
this evaluation. 

Your further participation in this study is of importance to us, and 
we hope you can assist SDC once more. Thank you again for your 
cooperation. 



Sincerely, 

(Mrs.) Bety Rasmussen 
Office of Program Planning 
and Evaluation 
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permitted us to increase significantly the number of evaluation questionnaire 
mailings. User Evaluation Questionnaires were mailed to all 1,837 user 
respondents to the first-wave mailing (i.e., prior to any followup mailing) of 
the Screening Questionnaire. This number far exceeded the originally antici- 
pated 875 mailings and helped to increase the likelihood of identifying readers 
of specific documents in the product sample. 

Because the number of non— user respondents was small, it was decided that all 
361 (again, excluding followup return respondents), not the originally planned 
subsample, would be surveyed for their potential interest. One exception to 
this decision was a group of approximately 75 respondents whose names were 
obtained from clearinghouse lists or EDRS, but who indicated that they had 
no familiarity with products of any NCEC units. Although this kind of anomaly 
is not particularly surprising, it was felt that direct contact with these 
individuals might have clarified their responses. For example, it is possible 
that some of these respondents were not those actually sampled, but instead, 
were unidentified successors to individuals on the mailing lists. Since this 
type of followup was not possible within time contraints of the project, this 
group was not included in the evaluation survey. 

Document Assignment Process . The initial review of some 100 Q1 returns 
suggested the need to reconsider the methodology that had been planned for 
assigning documents: members of the project’s Advisory Board were to be convene 

for purposes of creating general user-group packages (i.e., a document set, to 
be used for each possible user group). The first Q1 returns indicated that 
interest areas and prior familiarity with a particular (related) NCEC unit 
did not always correlate, nor did either or both responses in these areas 
match expectations for given user groups. Since the extent of these exceptions 
was not known, a decision was made to individualize the document assignment 
process, using a process very much like selective dissemination of information. 

To support this assignment process, two tools (shown in Exhibits III— 5 and III-6 
were developed: 1) a Respondent Card, and 2) a Document Card. Each Respondent 

Card contained information transcribed from the Q1 returns concerning interest 
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EXHIBIT III-5. A RESPONDENT CARD FOR DOCUMENT ASSIGNMENT PROCEDURE 




Description of Elements: 

Mailing Label: pre-printed mailing labels, which included the user's code, 

were affixed. 

2. Q2 or Q3 : was circled to indicate which questionnaire respondent was to 

receive. 

3* User Class : staff-assigned general audience category from responses to 

questions 1 and 2. 

4. listing of NCEC units in order from Ql; for "user" respondents, 

"read/ skimmed" responses were circled. (These responses were "starting 
points" for the Q2 document assignment process.) 

5* [q3] * numbers representing areas of interest from question 3; respondents' 

answers were circled. (These were secondary clues for "user" respondents, 
and the primary clue for "non-users".) 

6* "Possible Assignments" box : used by staff to note document code numbers of 

possible assignments, by product type. GP assignments were taken from 
"general interest" document group. 

7* "Documents Assigned" box : used to note document codes, with bibliographies 

listed first and marked (B) . Asterisks were used to indicate documents that 
a respondent was known to have requested (from clearinghouses) or purchased 
(from EDRS) . [This part of the card became the record for packaging each 
respondent's Q2 or Q3 . I 



EXHIBIT II 1-6. A DOCUMENT CARD FOR DOCUMENT ASSIGNMENT PROCEDURE 




RIC PRODUCTS 1969-1970 




ED04I5W LI 001 889 1 

ERIC Products I W?- If 70; A tiMtagraphy of In- I 
formation Analysb PatlkaU — » St Ik* ERIC 
Cfearinghuuvs July 1969 -Jimm 1979. 

ERIC Ciouringhouso on Library and Intormation 
Sciences. Washington. D C 
Spon* Agency— Educational Resources Informa- 
tion Center, DIR 
Pub Date Aug "’ll 
Note— i7p 

EDRS Price iU -SO.25 HC-S2.4S i 

Descriptors— Bibliographic Citations, •Ribliogra- 
phies, •Clearinghouses. 'Education, Educa- 
tional Research. Educational Ratourtat, Infor- 
mation Dissemination, InhKiMtion Sarvices, 
•Information Systems, Umsts tt Reviews, 
•Pub lications. Research Mie s (MMcaissi) 

The third annual sag^h, «r non 
Clearinghouse publications reflecis i nhu mat ion ' 
analysis activities of each Clearinghouse, ft in- 
eludes til substantial bibliographies, rttitw j 
papers and statc-of*the«srt papers beatified at j 
ERIC publications. The 366 annotated items for / 
Fiscal Year 1970 are arranged alphabetically by f 
Clearinghou^ and, within each Clearinghouse, 
alphabetically by author, The availability and cost 
of each cited document is provided. This issue 
also has an alphabetic subject index using ERIC 
Thesaurus terms. The first issue of ERIC 
Products for Fiscal Year 196$ is available as 
ERIC document £0 029 161 usd the second, for 
Fiscal Year 1969, as ERIC document CD 034 
089. (NH) ■ 





Description of Elements: 

1. Brief title 

2. Abstract from Research In Education 

3. SDC-assigned code number 

4. Product-type class 

5. £3: interest areas from question 3 that were appropriate descriptors for 
the document. [In this case, the document was felt to be relevant to all 
respondents * interest areas.] 

6. Users : .listing of general audiences. [This bibliography was one selected 

for *.he '‘general interest" group of documents.] 
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areas and NCEC units with which respondents had indicated previous awareness. 

(These cards were also used as the control file for Q2/Q3 mailings.) Document 
Cards were created for each document in the product sample and contained the full 
abstract from Research In Education , numbers representing appropriate descriptors 
from the 34 interest areas listed in the third question of Ql, and an indication 
of the intended audiences. Document Cards were placed in two types of files, 
one organized by NCEC unit and the other, by each of the 34 interest areas. 

For users — those who were likely product readers — the 10 documents to be 
assigned* were selected in the following manner: 

1. Each Respondent Card and the respondent’s Ql return were studied to 
help in identifying the most appropriate documents from the 
clearinghouses or NCEC units that had been checked. 

2. Document Cards for each of these units were drawn and reviewed 

for potential relevance. Document codes were written on the Respon- 
dent Card, with some special notation by those considered 
potentially most relevant or \ appropriate. 

3. If more than 10 documents were identified, the most relevant 
were selected or a random selection was made. 

4. If fewer than 10 documents had been identified at this point, 
the range of possible candidates was increased by identifying 
documents from other-than-check^d NCEC units. (These 
documents were included for potential interest evaluation, 
since the likelihood was slight that they would have been read.) 

5. As a last recourse for completing the package of 10 documents, 
selections were made from a group of 12 general interest 
documents, such as ERIC Products 1969-1970 . 

Although there were similarities in the packages developed for the various users, 
the differences in respondents’ combinations of interests resulted in truly 
individualized packages. 



One group, the information intermediaries, presented an interesting set of 
assignment decision points. In many cases, these respondents checked all 34 
interest areas. The approach in developing packages for these individuals 
was to identify documents seemingly relevant to their own professional growth 



The procedure for assigning documents to non-users was the same, beginning at 
Step 4. 
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or interest (e.g., in Library and Information Sciences) ^nd those materials 

which were ref erence-type materials. For' example, with ar school librarian, 

*. - * 

Bibliographies and Practical Guidance Papers that append to be sources of 
intructional materials were often selected. 



Examples of the documents assigned for a few selected respondents are shown 
in Table III-13. User Profile columns illustrate responses to each question 
in Q1 . The last column represents SDC code numbers l^^^o'&cuments assigned. 

w 

Mailings /Returns . A total of 1,837 survey participants included in the Q2/Q3 
mailings translated into a total of 18,370 questionnaires , since each respondent 
received a total of 10 different questionnaires with document representations 
attached individually to each. The Respondent Card described earlier served as the 
record for this production effort of identifying correct document representations 
for each respondent’s package. (Each document representation, as shown in 

Appendix D, included the SDC-assigned document code.) 

A complete mailing package consisted of a cover letter, 10 questionaires— 

each coded and sequentially numbered— and a pre-paitt^eturn envelope. A 
postcard was used for the followup mailing. Returns to this mailing by samples 
and user groups are shown in Tables III-14 and IlT-15. Although no control was 
exerted over the composition of the Q2/Q3 mailings by user groups, the tables 
have been developed to reflect this information in order to identify the con- 
tributers to the 68 percent and 71 percent return rates. 

Editing and Keypunching . Of the 1,559 questionnaire packages returned, 

43 were not included in the data analysis because identifying data 
(i.e., respondent code), had been removed or there was evidence that an 
individual' other than the survey participant had completed the questionnaires. 

The questionnaire returns for 1,251 (Q2) and 255 (Q3) respondents were key- 
punched and entered in the evaluation data file for analysis. In some 
instances, the full 10 evaluations for each respondent could not be included 
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TABLE III-13 



EXAMPLES OF DOCUMENTS ASSIGNED TO SELECTED RESPONDENTS 



User Profiles (from Q1 data) 


Documents Assigned 


q . 1 


q . 2 


q. 3 


q. L 




Code/Nusbers 


(Setting) 


(Role/ function) 


(Interest Areas) 


(Products 




(^successful matches 




Raad/Sktmed) 




with a 


product reader) 


State Education 


Program 


English 


PREF 


l. 


RE004 


6. 


RE002* 


Agency 


Special 1st 


Compensatory Educ. 


EMC 


2. 


RE017 


7. 


RE005* 


[from Random 




Curriculum Devei * 


RE 


3. 


BB002 


8, 


TE023 


Sample] 




Early Childhood Educ. 


TE 


4. 


TE008* 


9. 


TE033* 






Instructional Mat. 

Reading 

Sec. Educ. 

Elem. Educ. 

Teacher Educ • 




5. 


TE011* 


10 . 


PP002* 


Eie*d. School 


Teacher 


Elem. Educ. 


FL 


1 . 


fit' 04 


6 . 


RE005* 


[from Random 




Read ing 






RE004* 


7. 


FLO 12 


Sample] 




Tests and 




3. 


RE017* 


8. 


FL022 






Measurements 




4. 


FLO 16 


9. 


RE002* 










5. 


FLQ19 


10. 


PP002 


School District 


Associate 


Curriculum Devei. 


PREP 


1 . 


ECG18* 


6 . 


SPQQ3* 


Central Office 


Superin- 


Compensatory Educ. 


CG 


2. 


SP032* 


7. 


TM001* 


[from Non-Random 


tendent 


Early Childhood Educ. 


PS 


3. 


RE032* 


8. 


PP016* 


Sample] 




Elem. Educ. 


EM 


4. 


EM004* 


9. 


PP012* 






R & D 


EC 


5. 


RE005* 


10. 


ppoo:* 






Sec. Educ. 


LI 
















RE 
















SP 
















TM 










Other: Adult 


Teacher 


Early Childhood Educ. 


EMC 


1 . 


BB004 


6 , 


PS012 


Education 




Adult & Continuing 




2. 


PS007 


7. 


AC018 


[from Non- 




Education 




3. 


ACOOf 


8. 


AC020 




Adult Basic Educ. 




4. 


psgu: 


9. 


AC022 


Random Sample] 




Psychological Serv. 




5. 


AC0D9 


10. 


AC027 


College/ 


Media Coor- 


Adult Education 


EMC 


X. 


EMQ01 


6, 


HE011 


University 


dinator or 


Curriculum Devei. 


EM 


?. 


EM0X6 


7. 


HE020 




instructional 


Educational Madia 


HE 


3. 


BB005 


8. 


SP008 


[from Non- 


technology 


& Technology 




4. 


EM004 


f , 


SP002* 


Rando© Sample] 


specialist 


Higher Educ, 
Instructional Mat. 
Teacher Educ. 




5. 


EM01 % 


10. 


SP0 32 
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TABLE III- 14. USER EVALUATION QUESTIONNAIRE (Q2) MAILINGS AND RETURNS 
BY USER GROUPS AND BY SAMPLES 





RANDOM SAMPLE 


NON-RANDOM SAMPLE 


TOTALS 


Mailed 


Returned 


Mailed 


Returned 


Mailed 


Returned 


Reading 

Specialist 


10 


5 (50%) 


23 


17 (74%) 


33 


22 ( 67%) 


Special 

Educator 


9 


0 (0%) 




23 (70%) 


42 


23 (55%) 


Vocational 

Educator 


S 


2 (25%) 


37 


28 (76%) 


45 


30 (67%) 


Supervisor of 
Instruction 


10 


4 (40%) 


54 


37 (69%) 


64 


41 (64%) 


Counselor, 

Psychologist 


25 


15 (60%) 


51 


33 (65%) 


76 


48 (63%) 


Researcher 


72 


45 (63%) 


i4Q 


101 (72%) 


212 


146 (69%) 


Instr. Resources 
Specialist 


38 


25 (66%) 


161 


123 (76%) 


199 


148 (74%) 


Program 

Specialist 


82 


55 (67%) 


216 


152 (70%) 


298 


207 (69%) 


Principal, 

Asst. Principal 


20 


14 (70%) 


67 


44 (66%) 


87 


58 (67%) 


Elementary 

Teacher 


14 


8 (57%) 


24 


17 (71%) 


38 


25 (66%) 


Secondary 

Teacher 


25 


11 (44%) 


55 


45 (82%) 


80 


56 (70%) 


College 

Professor 


27 


12 (44%) 


334 


229 (69%) 


361 


241 (67%) 


College 

Administrator 


13 


8 (62%) 


88 


65 (74%) 


101 


73 (72%) 


Superintendent , 
Asst. Super, 


15 


11 (73%) 


63 


45 (73%) 


78 


56 (72%) 


Other Admin. 
Position 


12 


5 (42%) 


30 


21 (70%) 


42 


26 (62*0 


Unclassified 


6 


2 (33%) 


/ 5 


49 (65%) 


81 


51 (63%) 


TOTALS 


386 


222 (58%) 


1451 


1029 (71%) 


1837 


1251 (68%) 



TABLE III- 15. NON-USER EVALUATION QUESTIONNAIRE (Q3) MAILINGS AND RETURNS 
BY USER GROUPS AND BY SAMPLES. 



* 


RANDOM SAMPLE 


NON-RANDOM SAMPLE 


TOTALS 


Mailed 


Returned 


Mailed 


Returned 


Mailed 


Returned 


Reading 

Specialist 


8 


7 (88%) 


3 


1 (33%) 


11 


8 (73%) 


Special 

Educator 


6 


4 (67%) 


1 


1 (100%) 


7 


5 (71%) 


Vocational 

Educator 


14 


9 (64%) 


1 


0 (0%) 


15 


9 (60%) 


Supervisor of 
Instruction 


5 


4 (80%) 


0 


0 (0%) 


5 


4 (80%) 


Counselor, 

Psychologist 


22 


12 (55%) 


5 


3 (60%) 


27 


15 (56%) 


Researcher 


18 


11 (61%) 


5 


3 (60%) 


23 


14 (61%) 


I"str. Resources 
Specialist 


24 


15 (63%) 


8 


7 (88%) 


32 


22 (69%) 


Program 

Specialist 


21 


16 (76%) 


13 


10 (77%) 


34 


26 (76%) 


Principal , 

Asst. Principal 


15 


12 (80%) 


5 


4 (80%) 


20 


16 (80%) 


Elementary 

Teacher 


23 


17 (74%) 


3 


3 (100%) 


26 


20 (77%) 


Secondary 

Teacher 


60 


49 (82%) 


18 


11 (61%) 


78 


60 (77%) 


College 

Professor 


20 


13 (65%) 


15 


11 (73%) 


35 


24 (69%) 


College 

Administrator 


13 


9 (69%) 


2 


2 (100%) 


15 


11 (73%) 


Superintendent , 
Asst. Super. 


3 


3 (100%) 


4 


3 (75%) 


7 


6 (88%) 


Other Admin. 
Position 


6 


3 (50%) 


6 


2 (33%) 


12 


5 (42%) 


Unclassified 


9 


6 (67%) 


5 


4 (80%) 


14 


10 (71%) 


TOTALS 


267 


190 (71%) 


94 


65 (69%) 


361 


255 (71%) 
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because some questionnaires were unusable. Of the total of 12,510 question- 
naires, only 57 of the returned questionnaires (distributed over that many 
respondents) were lost to the sample because of packaging errors such as the 
inclusion of wrong or duplicate document representations or unanswered question 
naires . 

Responses to the User Evaluation Questionnaire required a minimum of editing. 
One typical problem was multiple responses to a single-response question, e.g. 
the respondent who indicated that a document was "relevant" (now) but "somewhat 
(formerly). In these cases, the more positive response was taken. 

The final question concerning respondents’ willingness to be product evaluators 
in future evaluations was hand tallied. Responses to open-ended questions and 
general comments were recorded and are reported in Volume II of this report. 
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THE SPECIALISTS' SURVEY 



Each of the 242 specialists was contacted initially by mail with the letter 
shown in Exhibit III-7. This Introduction to the request for their partici- 
pation was followed up by a telephone call from one of the project staff 
members. Upon receiving an indication of the specialists* willingness to 
participate, staff members then discussed the appropriateness of the subject 
areas from which documents had been assigned. Each specialist was notified 
that $15, a token honorarium, would be paid for each evaluation. 

The results, from point of contact through return, are traced in Table III-16. 

TABLE III-16. CONTACTS AND RET' US EOF THE SPECIALISTS' SURVEY (Q4) 




To the extent possible, substitute evaluators were found (among participants 
and those not selected originally) to cover the "lost" document assignments. 

The high return figure also resulted from several prompting efforts, bv followup 
postcard and telephone calls. This number of participants translates into 
two evaluations for 26 documents and three evaluations for 127 documents.* 

It is necessary to recall that document assignments were made at the time 
specialists were sampled. However, some changes in these initial assignments 




ft 

This total is greater than 146 because, for the Specialists' Survey, parts of 
two different documents were treated as separate volumes. The four separate 
PREP reports on Treating Reading Difficulties were evaluated separately, as 
were the five individual chapters (each of which represented a state-of-the-art 
paper in a different area) of Social and Technological Changes? Implications 
for Education . 
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EXHIBIT III-7 , LETTER OF INVITATION TO SPECIALISTS 

(in reduced form) 



SDC SYMtM DlVllOPMlNI ('( )KI'( )KAI l( )N 

C ti'umhii F jIIs l him h V JJtMl 



Dear 

You have been recommended to us by one or more of your colleagues 
who serve as Advisory Board Members to ERIC (Educational Resources 
Information Center) Clearinghouses. We approached over 200 of these 
educational leadeis to help us in identifying a c ross-sec t ion of 
specialists who would be qualified and interested in helping us 
evaluate selected educational publications. During the next few 
weeks we will Contact you by phone to determine vour interest in 
participating in this evaluation project. 

System Development Corporation, under contract to the U.S. Office of 
Education, Is nonducting an evaluation study of selected information 
analysis products orepared by the National Center for Educational 
Communication in USOE. In the attached Project Description, several 
aspects of the study objectives and design are described. The sample 
of 150 documents selected for evaluation includes: 

t ERIC products (e.g., bibliogtaphles , state-of-the- 
art papers) 

• PREP (Putting Research Into Educational Practice) 
Reports 

• Educational Materials Center bibliographies 

Our study involves a General Field Survey of 5000 educators and a 
Specialists 1 Survey. It is in regard to the Specialists 1 Survey 
that we are seeking >our assistance. 

You, as a specialist, will be 3sked to evaluate from one to four 
documents, for which full text copies will b.> provided. We are 
asking that you examine the document (s) carefully and then complete 
a questionnaire that will provide us with an in-depth assessment 
of the document's quality. Our goal is to obtain evaluations for 
each document from three -d if ferent specialists. Although we plan to 
identify all participants in the Specialists' Survey, the evaluations 
will be kept confidential and responses will not be attributed to a 
particular individual* 

During our phone conversation with you, we will (a) determine vour 
interest in serving as a document evaluator, (b) verify tee appro- 
priateness of the document (s) selected, and (t * confirm the scheduling 
of, and remuneration for, your services. 

We look forward to your part ic ipat ion m this project- You will hear 
from one of our project staff members soon. 



JW/lsr 

Enclosure 



Sincerely , 



Judith Wanger 






Project Director 

NCEC Product Evaluation Project 
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were required, based on information provided in conversations with specialists 
and in instances where "willing participants" were assigned additional documents 
that had been originally assigned to those indicating an unwillingness to 
participate. 

Most specialists reviewed two or three documents, some reviewed one, and one, 
as many as seven. Complete mailing packages consisted of cover letters^ 
questionnaires for document, the full text of documents (either an EDRS hard 
copy or an SDC-produced photocopy) , a remuneration form, and a pre— paid return 
envelope. Sample copies of each form are presented in Appendix D. 

Specialists returns were tabulated manually. The richest data from the 
questionnaires were in the free-form responses, into which the specialists put 
a great deal of thought and effort. Their responses are summarized in 
Volume II of this report. 
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RESPONDENTS 



IV. DESCRIPTION OF THE RESPONDENT POPULATIONS 



Background information on survey respondents of the General Field Survey was 
collected only once, through the Screening Questionnaire (Ql). The level of 
detail was minimal, but sufficient to characterize respondents and to identify 
their interests for purposes of assigning documents in the Evaluation Question- 
naire Survey. These variables were carried forward for respondents participating 
in the Evaluation Questionnaire Survey. 



Three different respondent populations of the General Field Survey are described 

throughout Section A of this chapter. They are; 

• Screening Questionnaire (Ql) Respondents in the 
random and non-random samples 

• User Evaluation Questionnaire (Q2) Respondents in 
the random and non-random samples. (These respon- 
dents are referred to as users since they indicated 
in their Ql return that they had read or skimmed pro- 
ducts from at least one NCEC unit.) 

• Non-user Evaluation Questionnaire (Q3) Respondents 
in the random and non-random samples. (These 
respondents are referred to as non— users since 
they indicated no prior familiarity with NCEC 
products, or that they had only heard about them.) 

The random and non-random samples* are reported separately in each table so that 

comparisons between these two groups can be made. The dual approach in 

sampling is an important element in the objective of developing a recommended 

procedure for identifying future product evaluators, and comparisons of their 

representations of the educational community are quite instructive. However, 

for the later analysis and reporting of evaluation data (in Chapter VXI) from 

the User and Non-user Evaluation Questionnaires, the two samples are combined. 

(The rationale for this decision is discussed in Chapter VI.) 




Background information concerning respondents to the Specialists r Survey was 
obtained from a special form attached to the survey instrument. They were 
asked only to provide their institutional affiliation and position. 

* For a review on the- composition of these two groups, see Appendix G, 
Glossary of Terms. 
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A. 



GENERAL FIELD SURVEY RESPONDENT POPULATIONS 



Three of the four items on the Screening Questionnaire were structured to obtain 
data on the respondents' work settings, roles or functions in the educational 
community, and subject areas of interest. Complete questionnaire items are 
contained in Appendix D; the stems of the three questions pertinent to this 
discussion are repeated here for reference: 

• In what institutional setting are you working? (Check as 
many as apply.) 

• What is your main professional role or function in the 
educational community? (If you have a dual assignment, 
please identify your principal role by placing a "1" before 
the single applicable item below and a "2" before the secondary 
role . 

• Which of the following areas represent your major professional 
interests? (Check as many as apply.) Please circle the 

one area that represents your principal professional interest . 

The distributions of responses to these questions among the Screening 
Questionnaire Respondent Population, and later among Evaluation Questionnaire 
respondents, are described below. 

1. SCREENING SURVEY (Ql) RESPONDENTS 

Background Variables . As reported in Chapter III, multiple responses to the 
first two questions were eliminated through an editing process. The single 
most appropriate setting was selected to match the principal role or function 
identified by the respondent. (For cases in which respondents did not indicate 
a principal role among several, the staff selected roles which appeared to 
carry higher "information stress.") 

The respondent population size (3,013) was too small to allow the mapping of 
responses to the two questions on setting and role/ function onto a matrix. In 
most cases, the size of the population at each reasonable intersection (e.g., 
elementary school with reading specialist) was inadequate to support any stable 
percent aging . Therefore, Tables IV— 1 and IV~2 separately report responses by 
work setting and by role/function. 
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TABLE IV- 1. 



WORK SETTINGS OF SCREENING SURVEY (Q2) RESPONDENTS 
BY SAMPLES 



SETTINGS 

Elementary 

School 

Junior High/ 

Middle School 

Senior /Vocational 
High School 

Junior College/ 
Community College 

Four-Year College/ 
University 

School District 
Central Office 

State Department 
of Education 

Non-Profit 

Organization 

Other Settings 



RANDOM SAMPLE 
[N=934] 1 


NON-RANDOM I 

SAMPLE 
[N= a 2 > 079] 


TOTALS 

[N-3,013] 


No . % I 


No • % 


No. % 


137 


(15%) 


175 


( 8%) 


312 


(10%) 


119 


(13%) 


75 


( 4%) I 


194 


( 6%) 


147 


(16%) 


164 


( 8%) 


311 


(10%) 


70 


( 8%) 


147 


( 7%) 


217 


( 7%) 


159 


(17%) 


I 772 


(37%) 


931 


(31%) 


60 


( 6%) 


I 260 


(12%) 


320 


(11%) 


165 


(18%) 


207 


(10%) 


372 


(12%) 


59 


( 6%) 


73 


( 4%) 


132 


( 4%) 


18 


( 2%) 


206 


(10%) 


I 224 


( 7%) 
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The distribution of respondents by work settings is given in Table IV— 1. 

Within the random sample, the distribution of respondents in the first seven 
settings closely parallels the proportions of the originally sampled popul- 
ation. Although a detailed analysis of non-respondents was not performed, 
the proportionality shown in this and subsequent tables (plus the evidence 
of comparable return rates for each sampling unit) suggests that no major 
biases are present in this population. 

Comparison of the distributions between samples shows selected cases of under- 
and over-representations. For example, in the four-year college and university 
setting, 37 percent in the non-random sample far exceeds the 17 percent in the 
random sample; this difference may actually be even greater because the 17 
percent no doubt includes some USOE-support ed facility researchers who reported 
their settings as universities rather than non-profit organizations. 

On the other hand, the 20 percent representation of local school personnel 
(elementary, junior, and senior high school) in the non- random sample is only 
half that in the random sample (44 percent). This is not entirely surprising, 
given the relatively greater information-use patterns of university personnel 
and the more direct link of clearinghouses to universities. 

The relatively high number of non-random sample respondents in "Other Settings" 
reflects a range in both educational and non-educat ional settings that was not 
targeted in the sampling units selected for identifying the random sample 
population. Illustrative of these "Other Settings" are: state or local 

government agencies, state prisons, hospitals, public libraries, adult educa- 
tion facilities, and industries. 

The data in Table IV-2 show that differences in roles/functions between the 
two samples are not nearly as striking as for work setting. The large number 
of university-based respondents is most likely distributed among several 
categories, and those in professor/ researcher roles often chose the former 
as their principal role. Despite this, the relatively low percentage of 
researchers in the non-random sample contradicts some widely-held assumptions 
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TABLE IV- 2, ROLES/FUNCTIONS OF SCREENING SURVEY (Q2) RESPONDENTS 
BY SAMPLES 



* 

ROLES /FUNCTIONS 


RANDOM SAMPLE 
[N-934] 


NON-RANDOM 

SAMPLE 

[N-2,079] 


TOTALS 

[N-3,013] 


No. 


% 


No. 


% 


No. 


% 


Teacher/ 

Professor 


331 


(35%) 


695 


(33%) 


1026 


(34%) 


Reading 

Specialist 


21 


( 2%) 


38 


( 2%) 


59 


( 2%) 


Superintendent/ 
Asst. Super. 


27 


( 3%) 


100 


( 5%) 


127 


( 4%) 


President/ Vice 
Pres . /Dean 


22 


( 2%) 


112 


( 5%) 


134 


( 4%) 


Principal/Asst. 

Principal 


56 


( 6%) 


106 


( 5%) 


162 


( 5%) 


Supervisor of 
Instruction 


25 


( 3%) 


76 


( 4%) 


101 


( 3%) 


Personnel/ Counselor/ 
Psychologist 


67 


( 7%) 


82 


( 4%) 


149 


( 5%) 


Program Specialist/ 
Consultant 


128 


(14%) 


294 


(14%) 


422 


(14%) 


Librarian/Media 

Specialist 


89 


(10%) 


234 


(11%) 


323 


(11%) 


Researcher /R&D 
Staff Member 


115 


(12%) 


166 


( 8%) 


281 


( 9%) 


Other Admin. 

Roles/Functions 


40 


( 4%) 


105 


( 5%) 


145 


( 5%) 


Other Roles/Functions 


13 


( 1%) 


71 


( 4%) 


84 


( 3%) 



concerning the characteristics of ERIC users and the composition of mailing 
lists. The seemingly low response from counselors/student personnel workers 
(non-random sample) is attributable to the unavailability of a mailing list 
from the ERIC Clearinghouse on Counseling and Personnel Services for this 
study. 

User Typology . Further analyses of the Screening Questionnaire were performed 
separately against each of these background variables; the results are con- 
tained in tables in Appendix F. However, to allow further analyses by user 
&J-2H P , anc * not against two different background variables, a typology was 

created from the two basic respondent characteristics — work setting and 

* / 

role/ function. 

The derivation of this typology is shown in Table IV-3. The first eight user 
groups were created as special-interest groups, regardless of setting. The 
remaining groups are setting-specific. For example, the category and number 
of Reading Specialists is the same as shown in Table IV-2; this is also true 
with Supervisors, Counselors, Researchers, Instructional Resources Specialists, 
and Program Specialists. Special Educator and Vocational Educator groups were 
created from respondents who indicated a primary interest (from question 3 in 
Ql) in one of these two priority arnas. In the bottom half of the table, roles 
are directly linked to settings; for example, the respondents in the Teacher/ 
Professor category from Table IV-2 are, in the typology, represented in one 
of several setting-specific teaching roles (e.g., elementary, secondary, etc.) 
unless they are included in one of the earlier "specialist" categories. If a 
respondent could be classified in two or more typology categories, a choice was 
made in favor of the category believed to be a more specific professional role. 

i nterests . An overview of the range of interests* reported by the two sample 
groups is displayed in Table IV-4. For reporting purposes, the one principal 
response circled by each respondent is designated "Major"; multiple responses 

The rationale for selecting these particular interest areas is discussed 
in Chapter III. 
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TABLE IV-3 . DERIVATION OF GENERAL USER TYPOLOGY 



GENERAL USER GROUP 


RESPONSES TO QUESTIONS IN SCREENING 
QUESTIONNAIRE. (Ql) 


RANDOM 

SAMPLE 

[N*=934 ] 


NON- RANDOM 
SAMPLE 
02079} 


Reading Specialists 


"Reading specialist" 


21 ( 2%) 


38 ( IX) 


Special Educators 


Principal interest in special educa- 
tion 


30 ( 31) 


39 ( 2%) 


Vocational 

Educators 


Principal interest in vocational 
education 


44 ( 5%) 


39 ( IX) 


Supervisors of 
Instruction 


"Supervisor of instruction" 


23 ( 3%) 


76 ( 4%) 


Counselors * etc. 


"Student personnel worker or 
guidance counselor," "psychologist" 


67 ( 7%) 


82 ( 4%) 


Researchers 


"Researcher or R&D staff member" 


115 (12%) 


166 ( 8%) 


Instructional 
Resources Spec. 


"Librarian, etc. and "media 
coordinator, etc." 


89 (10%) 


234 (11%) 


Program 

Specialists 


"Program specialist, consultant or 
coordinator" 


128 (14%) 


294 (14%) 


Principals in 
Elem. /Sec. 
Settings 


All Elem. /Sec. settings and 
"principal, asst, principal" 


51 ( 5%) 


100 ( 5%) 


Elementary 

Teachers 


Elementary and Preschool settings 
and "teacher" 


55 ( 6%) 


51 ( 2%) 


Secondary 

Teachers 


Secondary settings and "teacher" 


138 (15%) 


102 ( 5%) 


College 

Professors 


College settings and "professor" 


70 ( 77.) 


463 (22%) 


College 

Administrators 


College settings and 
"president," "dean," or "other 
administrative position" 


34 ( 4%) 


136 ( 7%) 


Superintendents 


School district and state department 
settings and "Superintendent, asst, 
superintendent" 


25 ( 3%) 


98 ( 5%) 


Other LEA-SEA 
Admin. Positions 


School district and state department 
settings and "other administrative 
position" 


23 ( 2%) 


40 ( 2%) 


Otherwise 

Unclassified 


Includes "school board members" and 
"others" in question 2 


19 ( 2%) 


121 ( 6%) 
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(all other areas checked) referred to as "Other." As substantiated by the 
tabulated data, two patterns are particularly evident: .1) a fairly even and 

comparable distribution of "Major" interests throughout both samples, and 
2) a greater breadth of "Other" interests in the non-random sample. 

The lack of Major" interest in eight areas in one or both samples is not 
viewed as a deficiency in the sample. Most of these categories represent 
special populations or environments that, as secondary interest areas, are 
subsumed in an individual’s overall function. The representation of random 
sample respondents having major interest in Fine Arts is a result of sampling 
parameters intended to draw some unintended audiences into the study; no one 
clearinghouse directly covers this area. 

The highest percentages of "Other" interests for both samples are in Curriculum 
Development and Instructional Materials. Research/Development falls fourth 
in order of frequency for the non— random sample. Although the questionnaire 
did not provide for a respondent’s coordinating terms to express his interests 
(i.e., I am interested in curriculum development relative to elementary science), 
the order of highest percentages suggests some priority in respondents' needs 
or preferences in subject treatment. For example, it appears that information 
for curriculum development is of more interest than research development. 

Patterns by user groups for both Major’ and "Other" interests are displayed in 
Tables IV-5 and IV-6, respectively. 

Percentages in Table IV— 5 do not total 100 percent across a row for each sample 
group (R for random and N for non-random) because some respondents failed to 
indicate a single major interest area. In the random sample, only 90 percent 
of the Reading Specialist respondents is included; in the non— random sample, 
only 73 percent of this group is included. Because of che way in which the 
typology was created, two groups — Special Educators and Vocational Educators — 
correlate 100 percent with the direct matches in major interest areas. 
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TABLE IV- A . INTERESTS — "MAJOR" AND "OTHER"— OF SCREENING SURVEY (Ql) 
RESPONDENTS BY SAMPLES 



INTERESTS 


RANDOM 


[N=934] 


NON -RANDOM {N*2079} 


Percent 
Ma j or 


Percent 

Other 


Percent 

Major 


Percent 

Other 


Adult/Continuing Educ. 


1% 


122 


2% 


21% 


Adult Basic Educ. 


0 


7 


0 


9 


American Indian Educ. 


0 


5 


0 


6 


Bilingual Educ. 


0 


6 


0 


11 


Compensatory Educ. 


0 


7 


1 


15 


Counseling /Personnel 


7 


12 


3 


14 


Curriculum Development 


2 


28 


4 


44 


Early Childhood Educ. 


2 


16 


4 


21 


Educ. Administration 


7 


15 


8 


27 


Media/Technology 


*> 


15 


3 


2b 


Elementary Educ. 


7 


21 


3 


26 


English/Language Arts 


0 


11 


2 


14 


Ethnic /Minor ity Educ. 


0 


15 


1 


22 


Exceptional Children 


5 


12 


3 


17 


Fine Arts 


3 


6 


0 


8 


Health/Safety/Driver Educ. 


1 


6 


1 


6 


Higher Educ. 


2 


12 


5 


22 


Home Economics 


1 


2 


1 


5 


Instructional Materials 


1 


23 


1 


35 


Junior Colleges 


1 


9 


2 


14 


Languages /Linguistics 


1 


6 


4 


11 


Library/Info. Sciences 


6 


7 


6 


13 


Mathematics Educ. 


2 


8 


1 


11 


Physical Educ. 


2 


5 


0 


6 


Psychological Services 


1 


8 


1 


13 


Reading 


3 


18 


3 


20 


Research/Development 


3 


18 


4 


30 


Rural/Small Schools 


0 


5 


0 


9 


Science Educ. 


3 


8 


2 


12 


Secondary Educ. 


2 


21 


1 


25 


Social Science Educ. 


4 


9 


2 


14 


Teacher Educ. 


1 


21 


6 


34 


Tests/Measurement 


3 


21 


2 


29 


Vocational/Technical 


6 


13 


4 


18 


Other Interests 

f 


3 


3 


3 


5 



T 



IV-9 

(page 10 blank) 






R 


N 


R 


N 


R 


i 

N 


R 


N 


R 


H 


R 


K 


R 


N 


R 


H 


R 


N 


R 


H 


R 


H 


R 


N 


Reading Specialist 


















0 


6 






















10 


0 






Special Educator 


















































Vocational Educator 
































— — — 


















Supervisor of 
Instruction 


4 


0 














4 


1 






4 


17 


0 


4 


0 


5 


4 


1 


16 


4 


0 


5 


Counselor , 
Psychologis t 


0 


1 


















70 


45 






0 


2 


0 


4 


0 


8 










Researcher 


i 


1 










1 


0 






2 


1 


5 


1 


5 


4 


6 


4 


1 


2 


0 


1 


2* 


i o 


Instructio \al 
Resources Specialist 
























i 


0 


1 


1 


0 


0 


, 2 


IQ 


16 


4 

* 


3 


0 


1 


Program Specialist 


2 ! 


4 


1 


0 






0 


1 


2 


3 


7 


3 


5 


8 


2 


5 


2 


6 


4 


3 


1 


2 


5 


2 


Principal, 

Asst. Principal 


Q 


1 








1 










0 


2 


4 


11 


0 


11 


39 


25 






27 


17 






Elementary Teacher 










0 


2 










2 


0 


■> ! 


4 


. 11 


23 


0 : 


4 






66 


28 


0 


/* 


Secondary Teacher 










0 


1 


0 


4 


0 


1 


2 


0 


1 


3 


1 


0 


2 


0 


I 


1 


1 


1 


9 


10 


College Protessor 


1 


2 


















0 


3 


0 




0 


5 


7 


4 


3 


•> 

L 


1 


2 


* 

X 


2 


College* Administrator 


6 


4 














0 


1 


6 


3 


3 


4 


0 


1 


13 


11 






0 


1 


0 


1 


Super i n ten den t , 
Asst. Super. 










0 


! 














4 


18 


i« 


1 


80 


43 






0 


2 






0 trier Admin, 
Position 


9 


i 


. 








4 


0 


0 


i 

i 

j 


> 




4 


5 






26 


20 


4 


3 


4 


3 
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TABLE IV- 5 



"MAJOR" INTERESTS OF SCREENING SURVEY (Ql) RESPONDENTS BY USER 
GROUPS AND SAMPLES (IN PERCENTAGES) . 







R 


N 


R 


S 


R 


N 


R 


N 


R 


N 


R 


N 


R 


N 


R 


H 


K 


N 


R 


N 


R 


N 


R 


N 


Reading Specialist 


5 


17 


5 


3 


0 


6 


0 


8 


0 


17 


5 


1 1 


15 


31 


35 


42 


5 


6 


15 


1 1 


75 


61 


20 


25 


Special Educator 


7 


0 


3 


13 


3 


0 


3 


8 


3 


33 


7 


21 


10 


18 


17 


38 


7 


26 


10 


23 


30 


31 


3 


K 


Vocational Educator 


32 


36 


11 


18 


? 


3 


7 


2 


0 


5 


3 


8 


23 


49 


0 


0 


9 


41 


9 


1 3 


> 


0 


0 


t 


Supervisor of 
Inst ruction 


12 


17 


8 


3 


4 


4 


12 


13 


16 


16 


4 


3 


72 


67 


28 


34 


40 


38 


16 


34 


24 


37 


12 


2 ( 


Counselor , 
Psychologist 


9 


18 


7 


5 


1 


6 


3 


10 


9 


11 


21 


38 


16 


28 


16 


20 


15 


24 


1 


10 


IB 


13 


3 




Researcher 


10 


18 


10 


7 


7 


4 


10 


10 


10 


19 


13 


16 


30 


36 


18 


23 


23 


34 


17 


27 


26 


18 


10 


* 


Instructional 
Resources Specialist 


11 


26 


6 


12 


6 


11 


3 


1 3 


2 


15 


8 


14 


22 


44 


13 


21 


3 


22 


42 


49 


26 


25 


19 


> ■ 


Program Specialist 


12 


21 


6 


13 


7 


11 


11 


1 3 


13 


17 


12 


15 


52 


46 


22 


21 


16 


33 


ih 


22 


32 


27 


7 


U 


Principal , 

Asst. Principal 


10 


9 


8 


n 


'•> 


6 


- 


8 


6 


8 


24 


22 


41 


59 


20 


26 


37 


38 


18 


20 


16 


38 


14 


2 ( 


Elementary Teacher 


0 


11 


2 


4 


4 


b 


4 


19 


9 I 


9 


4 


b 


7 


28 


34 


19 


0 


9 


4 


1 5 


25 


45 


20 


i 1 


Secondary Teacher 


10 


7 


5 


.1 


3 


6 


4 


1 3 


1 


4 


7 ; 


7 


15 


47 


\ 


1 


b . 


1 S 


h 


2 3 


1 


fa 


15 




Col lege Professor 


] 1 


2i 


\ 


8 


1 


5 


1 


8 


7 


12 


1 1 


(y 


26 


44 


1 i 


lh 




In 


14 : 


20 


i<>: 


24 


1 ; 


1 


College Administrator 


26 


29 


12 


8 


9 




12 




9 


12 


21 


17 


32 


46 


9 


12 


41 


41 


24 


26 


3 


15 


9 


c 


Superintendent, 
/ s' t . Super . 


20 


15 


12 


14 


12 


1 


8 


13 


16 


26 


24 


27 


44 


57 


32 


35 


12 


45 


24 


40 


48 


54 


16 


2 


Other Admin. 


" 


27 


9 


30 


13 


10 


13 


25 


13 


32 


26 


35 


22 


52 


13 


42 


30 


42 


13 


42 


17 


40 


9 


3 
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R 
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R 


H 


R 


N 


R 


N 


R 


N 


R 


H 


R 


N 


R 


N 


R 


N 


R 


N 


R 






11 


75 


61 


20 


25 


0 


14 


10 


44 


5 


( 


ft 


3 


0 


6 


0 


0 


15 


25 


0 


8 


5 


17 


0 


8 


5 


0 


0 


0 


5 


i 




23 


30 


31 


3 


10 


10 


23 


0 


0 


10 


0 


3 


0 


13 


3 


0 


5 


7 


23 


7 


0 


3 


13 


0 


5 


3 


3 


3 


0 


10 






13 


2 


0 


0 


0 


0 


8 


0 


5 


0 


3 


9 


3 


7 


23 


0 


5 


20 


28 


23 


33 


0 


3 


0 


5 


9 


0 


2 


0 


0 




i 


34 


24 


37 


12 


20 


24 


29 


12 


14 


0 


16 


4 


11 


8 


9 


4 


3 


40 


50 


4 


4 


8 


17 


4 


14 


8 


13 


4 


1 

7 


1 

4 






10 


18 


13 


3 


* > 

2 


M9 


22 


28 


32 


4 


2 


3 


1 


18 


30 


0 


2 


6 


12 


19 


21 


0 


1 


0 




4 


2 


3 


1 


40 


l 


1 


27 


26 


18 


10 


8 


20 


17 


6 


16 


5 


4 


3 


3 


17 


23 


3 


3 


20 


23 


9 


19 


10 


5 


10 


17 


10 


9 


2 


3 


10 


] 


7 


49 


26 


25 


19 


21 


16 


24 


11 


21 


7 


17 


3 


13 


13 


25 


*7 


12 


49 


57 


8 


19 


7 


17 


20 


36 


3 


20 


2 


15 


4 i 


■ 1 

^ , 




22 


32 


27 


7 


16 


17 


29 


12 


17 


6 


9 


7 


4 


9 


17 


1 


5 


30 


37 


7 


14 


9 


13 


6 


11 


8 


12 


5 


5 


9 


1 




20 


16 


38 


14 


20 


16 


19 


22 


23 


16 


16 


12 


6 


4 


5 


10 


6 


35 


39 


2 


3 


10 


5 


16 


13 


18 


14 


16 


13 


s 


1 




15 


25 


45 


20 


15 


11 


32 


14 


21 


9 


15 


2 


8 


0 


4 


0 


2 


11 


30 


0 


0 


4 


15 


4 


8 


1 18 


19 


5 


8 


0 






23 


1 


6 


15 


7 


10 


17 


9 


5 


6 


6 


9 


4 


4 


5 


3 


0 


12 


34 


1 


4 


8 


12 


0 


5 


5 


6 


7 


2 


1 






20 


lt> 


24 


15 


n 


14 


19 


10 


ii 


4 


2 


6 


2 


31 


34 


0 


2 


24 


32 


23 


S 

•1 5 


4 


7 


9 


5 


6 


6 


4 


1 


4 






26 


3 


15 


9 


9 


21 


18 


3 


10 


6 


8 


3 


5 


47 


44 


3 


4 


18 


29 


24 


22 


0 


9 
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14 


3 


11 


3 


8 


6 
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40 


48 


54 


16 


29 


16 


27 . 


24 


26 


16 


17 


12 


16 


8 


5 


12 


14 


24 


40 


4 


1 


8 


19 


12 


21 


20 


24 


12 


19 


24 




p 


42 


17 


40 


9 


32 


26 


20 


26 


35 


9 


25 


13 


20 


13 


15 


4 


22 


17 


32 


4 


13 


13 


32 


4 


27 


17 


35 


9 


25 


9 


2 
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TABLE IV-6 . "OTHER' 1 INTERESTS OF SCREENING SURVEY (QI ) RESPONDENTS BY 
GROUPS AND SAMPLES (IN PERCENTAGES). ' i 
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In Table IV-5, the reported interests provide further insight into the composi- 
tion of the respondent population. Assuming that in most cases major interest 
areas relate directly to roles/ functions or setting levels, the data contain 
no major surprises. For Secondary Teachers, there is evidence that foreign 
language teachers are a particularly well— represented group in the non— random 
sample, as are the social science/social studies teachers in both samples. 

The majority of respondents in the Instructional Resources Specialist group 
report major interest In Library and Information Sciences, rather than in 
Media and Technology, thereby clarifying the composition of this particular 
group . 



In Table IV-6, the totals are over 100 percent because of multiple responses by 
most respondents. Entries in larger print represent percent-ages, in either one 
or both samples, showing user group interest of 25 percent or more. The spread 
or narrowness of Interests seems to match reasonably well the homogeneity or 
heterogeneity of the user groups. Furthermore, within each interest area there 
is no discernable pattern of "zero user-group inte|j^g^.i ,, even in such areas as 

Fine Arts. It should be recognized, however, that -sc))^ of this breadth of 

•• ■ s s. ■' 

interest is probably attributable to roles/functiohs-bey.qpd those implied in 
the user group designations. For example, p^r^j^ges of secondary teachers 
interested in educational administration may rj^lect .^jjssponses from graduate 
students or from teachers whose goals include ^duo^tijo'^al administration. 
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The breadth pf differences between samples is most strikingly characterized in 
the interest index shown below. Distributions i ito three levels were 
made on the basis of numbers of interest areas checked by each ’ respondent 
(including the one circled area of major interest), The low index level 
includes respondents with two or fewer interest areas; medium, three to six 
interest areas; and high, seven or more interest areas. These cutoff points 
create the following distribution in each sample group: 



Interest 

Index 

Low 

Medium 

High 



Random 

[:^934: 

. 285 (315?) 

447 (48%) 

202 (22/Q 



Non-Random 
[N=2 ,079] 

364 (18%) 

906 (43%) 

809 (39%) 



O 

ERIC 



IV-15 









The two samples do not appear sufficiently dissimilar in their backgrounds to 
suggest differences in information needs stemming from roles/functions (except 
for the larger number of professors). If we continue to assume that the 
non-random sample is more "hooked in" to the educational information dissemin- 
ation system, then some reasons for these differences might be formulated. 

It could be that greater degree of contact with information sources is in 
itself a "need-creating" force. That is, the awareness and use of information 
may lead users into peripheral and/or related areas of reading. (In Chapter V, 
Table V-l, it is noted that the non-random sample is also "more aware" of 
products from different NCEC units than is the random sample.) The non-random 
sample could also be more accustomed to expressing interests in a variety of 
ways in order to retrieve required information. Less frequent information 
users may, on the other hand, relate the terminology of their interests more 
directly, and simply, to that of their roles/funct ions. 

Respondents’ Comments . More insight into a respondent population is often 
obtained through their responses to free-form questions and volunteered 
comments. In general, Q1 respondents were liberal in supplying unsolicited 
comments relative to ERIC, and to the NCEC products as a whole. The following 
comments are samples of compliments taken verbatim from letters or notes 
attached to returned questionnaires: 

• I could not effectively function without ERTC/Adult Education. 

• I am very interested in this report; feel teachers need 
to know and use these resources. 

• From the insert I have learned of reports that will be 
useful to me. Thanks for sending me this questionnaire. 

• I wish I were aware of these (products). 

• Would like to know much more about these. 

• It seems to me that all of them are very useful and 
interesting, regret that I have not seen or read any 
of them. 

• Use two clearinghouses’ products constantly. . .and EDRS. 
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• All are very helpful in performance of my tasks. . .thanks 
very much. 

• I am so unaware I suspect you of withholding information 
to keep me in ignorance. 

There were those that were less favorable, such as: 

• The clearinghouses don’t send me ERIC materials and I 
have written repeatedly. 

• I ordered ’’Introduction to ERIC” and felt the filmstrip/ 
record were not very good. 

• We have made an attempt to be included on ERIC 
dissemination material lists but have not been 
successful. 

• Permit me to make a suggestion: have GPO publish 

documents. $3.50 is too much for hard copy of 6-15 
page document and microfilm is not practical. Unless 
price is reduced, the products will collect dust on 
shelves. 

The questionnaire, particularly its color insert, prompted requests from 
survey participants for more information. Some 200 respondents requested 
copies of documents shown in the color inserts. Others asked to be included 
on mailing lists, or desired general Information on product availability. 
Interestingly, most of the requests were made by respondents who were already 
familiar with at least one NCEC unit. A general information sheet, prepared 
by NCEC for this study, was forwarded to these requesters. 



2. EVALUATION SURVEY (Q2 AND Q3) RESPONDENTS 

The typology created for the Q1 respondents was retained for the Q2 and Q3 
respondent file because no further request for background information was 
included in the second-phase Evaluation Questionnaire. 



User Evaluation Respondents . It should be recalled that Q2 was mailed to 1,837 
respondents who reported having read/skimmed NCEC products from one or more 
NCEC units (as detailed in Chapter III). The original questionnaire mailings 
represented a non-random to random ratio of approximately four to one; returns 
Increased that ratio to five to one. By user group, the proportions for the 
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total population of Q2 respondents (both samples combined) remained remarkably 
stable (see Table IV- 7) • The percentage remained the same in six cases, 
increased in five, and decreased in five. The most significant decrease noted 
was in the Secondary Teacher category. 

Non-user Evaluation Survey Respondents . All user groups are represented in 
the total respondent population of non-users as shown in Table IV-8. The 
composition of this population is generally proportionate to that of the 
User group (Table IV-7) with some few exceptions. Secondary Teachers comprise 
the largest group (24 percent) of non-users, but are among the least repre- 
sented in the user population. 

The ratio between samples is an artifact of the way in which survey partici- 
pants were selected. The random sample of non-users identified in the 
Screening Questionnaire was almost three times that for the non-random sample. 
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TABLE IV-?. DESCRIPTION OF USER EVALUATION SURVEY (Q2) RESPONDENTS 
BY USER GROUPS AND BY SAMPLES. 



USER GROUPS 


RANDOM 

SAMPLE 
[N=222 ] 


NON-RANDOM 
SAMPLE 
[N=1029 ] 


TOTAL 
[N=1251 ] 


Reading Specialist 


5 


( 2%) 


17 


( 2%) 


22 


( 2%) 


Special Educator 


0 


( OX) 


23 


( 2%) 


23 


( 2%) 


Vocational Educator 


2 


( IX ) 


28 


( 3%) 


30 


( 2%) 


Supervisor of Instruction 


4 


( 2%) 


37 


( 4%) 


41 


( 3 %) 


Counselor, Psychologist^ 


15 


( 7%) 


33 


( 3%) 


48 


( 4%) 


Researcher 


45 


(20%) 


101 


(10%) 


146 


(12%) 


Instr. Resources Specialist 


25 


(11%) 


123 


(12%) 


148 


(12%) 


Program Specialist 


55 


(25%) 


152 


(15%) 


207 


(17%) 


Principal, Asst. Principal 


14 


( 6%) 


44 


( 4%) 


58 


( 5%) 


Elementary Teacher 


8 


( 4%) 


17 


. ( 2%) 


25 


( 2%) 


Secondary Teacher 


11 


( 5%) 


45 


( 4%) 


56 


( 4%) 


College Professor 


12 


( 5%) 


229 


(22%) . 


241 


(19%) 


College Administrator 


8 


( 4%) 


65 


( 6%) 


73 


( 6%) 


Superintendent, Asst. Super. 


11 


( 5 X) 


45 


( 4%) 


56 


( 4%) 


Other Admin. Position 


5 


( 2%) 


i 21 


( 2%) 


26 


( 2%) 


Unclassified 


? 


( 1%) 


i 49 


( 5%) 


51 


( 4%) 
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TABLE IV- 8. DESCRIPTION OF NON-USER EVALUATION SURVEY (Q3) RESPONDENTS 

BY USER GROUPS AND BY SAMPLES 



USER GROUPS 


RANDOM 

SAMPLE 

[N-190] 


NON-RANDOM 

SAMPLE 

[N=65? 


TOTAL 

[N=255j 


Reading Specialist 


7 


( 4%) 


1 


( 2%) 


8 


( 3%) 


, Special Educator 


4 


( 2%) 


1 


( 22) 


5 


( 2%) 


Vocational Educator 


9 


( 5%) 


0 


( 02) 


9 


( 4%) 


Supervisor of Instruction 


4 


( 2%) 


- 


- 


4 


( 2%) 


Counselor, Psychologist 


12 


( 6%) 


3 


( 52) 


15 


( 6%) 


Researcher 


11 


( 6%) 


3 


( 52) 


14 


( 5%) 


Instr. Resources Specialist 


15 


( 8%) 


7 


(11%) 


22 


( 9%) 


Program Specialist 


16 


( 8%) 


10 


(152) 


26 


(10%) 


Principal, Asst. Principal 


12 


( 6%) 


4 


( 6%) 


16 


( 6%) 


Elementary Teacher 


17 


( 9%) 


3 


( 52) 


20 


( 8%) 


Secondary Teacher 


49 


(26%) 


11 


(17%) 


60 


(24%) 


College Professor 


13 


( 7%) 


11 


(172) 


24 


( 9%) 


College Administrator 


9 


( 5%) 


2 


( 32) 


11 


( 4%) 


Superintendent, Asst. Super. 


3 


( 2%) 


3 


( 5%) 


6 


( 2%) 


Other Admin. Position 


3 


( 2%) 


2 


( 32) 


5 


( 22) 


Unclassified 

* 


6 


( 3%) 


4 


( 62) 


10 


( 42) 
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SPECIALISTS' SURVEY RESPONDENT POPULATION 



Specialists were requested to supply information regarding their institutional 
affiJiation and position. Setting and role/function categories used to classify 
Specialists were essentially the same as those for the General Field Survey, 
as shown in Table IV-9. 

Of the 194 responding Specialists, 117 (60 percent) are associated with four- 
year colleges or universities. This exceptionally high, but not particularly 
surprising proportion is due to the method used in identifying Specialists. 

Of the 94 responding ERIC Clearinghouse Advisory Board members — who nominated 
specialists — institutional affiliations could be identified for 89; of these, 
approximately 50 percent are affiliated with four -year colleges or universi- 
ties. The inclusion of 22 percent from local and state settings is particularly 
encouraging . 



TABLE IV-9. 



DESCRIPTION OF SPECIALISTS’ SURVEY RESPONDENTS BY 
SETTING AND ROLE/FUNCTION 



ELEMENTARY SCHOOL — Principal 5 

JUNIOR HIGH SCHOOL — Assistant 

Principal 1 

SENIOR HIGH SCHOOL 

Teacher 5 

Assistant Principal 1 

Director 1 

SCHOOL DISTRICT CENTRAL OFFICE 

Supervisor of Instruction 1 

Consultant 1 

Director/Coordinator 6 

COUNT'’’ SCHOOL SYSTEM — Teacher 2 

STATE DEPARTMENT OF EDUCATION 

Assistant Superintendent 2 

Program Specialist/ 

Consultant /Coordinator 9 

Director /Assistant Director 9 

JUNIOR/COMMUNITY COLLEGE 

President 1 

Academic Dean 2 

Director 1 

Other Administrator 1 



FOUR- YEAR COLLEGE /UNIVERSITY 

Professor 

Academic Dean 

Student Personnel Worker 

Consultant /Coordinator 

Librarian 

Researcher 

Director/Assistant Director 
Other Administrator 

FEDERAL /STATE GOVERNMENT AGENCY 

Program Specialist 
Director 

NON-PROFIT ORGANIZATION/PRO- 
FESSIONAL SOCIETY OR ASSOCIATION 

President 

Program Specialist 
Researcher 

Director/Assistant Director 
Other 

COMMERCIAL ORGANIZATION 
Other Administrator 

OTHER 



90 

4 
1 

5 
2 
2 

12 

1 



3 

1 



2 

1 

4 

10 

1 



1 

6 



Total = 194 




IV- 2 2 



1 



(... O 






c 

c 

3 

3 



c 

C 



c 



c 

3 



r 

r 

c 



r 

m 

3 

r 

€ 

m 

* • 

* „ 
c 

m • 

C 





I 

c 

* 

# 

< 

f 

( 

( 

c 

m 

( 

* 

f 

< 

( 



r 

* 

c 

3 



€ 

«*< 

C 



t 



FAMILIARITY WITH PRODUCTS 



V. SURVEY FINDINGS ON RESPONDENTS* FAMILIARITY 
WITH NCEC INFORMATION ANALYSIS PRODUCTS 



The survey findings address the question of respondents* familiarity with NCEC 
products in two ways: 1) at a general level, from Screening Questionnaire Survey 

data, and 2) at a product-specific level, from the Evaluation Survey data. Inter- 
pretation of the data is particularly difficult because there are no baseline 
data or agreed-upon statements of expectations against which the findings can be 
compared. The first two sections present the findings of the General Field Survey, 
and in the final section, descriptive information concet::^ announcement and dis- 
tribution strategies used with the products are reported and discussed. 



A. RESPONDENTS* GENERAL FAMILIARITY WITH PRODUCTS 

The Screening Questionnaire (Ql) was designed to identify potential product 
evaluators, and it served as a source of data on respondents* general familiarity 

with NCEC information analysis products. The key screening item was: "Prior to 
receiving this questionnaire, had you seen or read an information product pre- 
pared by any one of these NCEC units?'* For each NCEC unit or product group 
(i.e., ERIC Clearinghouses or PREP reports or EMC bibliographies), three res- 
ponse choices were provided: 

• I have read or skimmed 

• I am aware of, but have not read or skimmed 

• I am not aware of 



Background information on the products, acronyms of the originating units, 
sponsoring agencies, institutional locations, and the color insert of sample 
products, were provided as additional recall stimuli. 



An over. Lew of the results, by NCEC unit, is displayed in Table V-l. Per- 
centages total over 100 because most respondents are represented more than 
one time in either or both columns, i.e., they might have read or skimmed 
products from one or more units but only heard about products from others. 
(These same respondents are also included in the response percentages for the 
"no awareness" choice, not shown in this table.) 
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TABLE V-l. PERCENTAGE OF PHASE 1 RESPONDENTS (GENERAL FIELD 
SURVEY) BY SAMPLE GROUPS, WHO REPORT HAVING READ/ 
SKIMMED OR HEARD ABOUT NCEC PRODUCTS. 





RANDOM [N=934] 


NON-RANDOM 


[N-2079] 




Percent 

Read/Skimmed 


Percent 
Heard About 


Percent 

Read/Skimmed 


Percent 
Heard About 


j PREP Reports 


13% 


12% 


20% 


13% 


EMC Bibliographies 


21 


18 


41 


17 


Adult Education CH 


7 


18 


17 


25 


Counseling/Personnel CH 


8 


18 


16 


24 


Disadvantaged CH 


9 


17 


25 


21 


Early Childhood CH 


12 


19 


23 


23 


Educ. Management CH 


9 


16 


21 


20 


Media/Technology CH 


11 


16 


26 


20 


Exceptional Children CH 


12 


17 


16 


24 


Higher Education CH . 


7 


15 


16 


23 


Junior Colleges CH 


5 


14 


14 


19 


Languages /Linguistics CH 


5 


15 


13 


22 


Library/Info. Sciences CH 


6 


17 


17 


22 


Reading CH 

V 


9 


17 


21 


20 


Rural/Small Schools CH 


4 


14 


15 


1 Q 


Science/Mathematics CH 


5 


16 


A k/ 

13 


21 


Social Science CH 


5 


18 


15 


21 


Teacher Education CH 


10 


16 


22 


21 


English CH 


9 


17 


16 


24 


Tests/Measurement CH 


16 


21 


22 


25 


Vocational/Technical CH 


10 


16 


20 


21 
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Percentages displayed in this table represent 72 percent of the total random 
sample population and 9ft percent of the non-random sample. Therefore, 2,626 
respondents, or 87 percent of the total population of 3,013, report some fami- 
liarity with at least one NCEC unit. As expected, the non-random sample is 
’’more aware,” with percentages in the read/skimmed column, on the average, approxi- 
mately 10 points higher and, in one case, as great as 20 points higher than 
those for the random sample. 

Typically, survey participants’ responses to this question represent some 

combination of all three response choices. To characterize this response 

pattern, a "familiarity index" was created to illustrate the differing levels 
* 

of awareness. A scale from one to three was used for each of the choice points: 

1 point, for "no awareness”; 2, for "awareness”; and 3, for "read/skimmed" . 

A maximum of 63 points was possible, representing a respondent who had read or 
skimmed products from all 21 NCEC units and product groups. Cutoff points were 
established to create three levels of awareness: low, 21 points (i.e., no aware- 

ness); medium, 22 to 29 points; and high, 30 or more points. Table V-2 displays 
the distribution of respondents in this index and demonstrates again that the non- 
random sample is considerably "more aware” that the random-sample population. 

TABLE V-2. SUMMARY INDEX OF SCREENING QUESTIONNAIRE (Ql) 

RESPONDENTS’ FAMILIARITY WITH NCEC PRODUCTS 





RANDOM SAMPLE 
[N=934] 


NON-RANDOM SAMPLE 
[N=2,079] 


Low 


263 (28%) 


123 (6%) 


Medium 


381 (41%) 


874 (42%) 


High 


290 (31%) 


1,082 (52%) 



Table V-3 displays percentages of respondents, by user group, who reported 
"read/skimmed" familiarity with products from each NCEC unit/product group. 
For example, 20 percent of Reading Specialists in the random sample (the R 
column), and 25 percent in the non-random sample (the N column), reported 
"read/skimmed" familiarity with PREP reports. 
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TABLE V-3 






R 


N 


R 


N 


R 


H 


R 


N 


R 


H 


R 


H 


R 


H 


R 


N 


R 


N 


R 


H 


R 


N 


R 


H 


Heading Specialist 


20 


25 


20 


28 


5 


3 


0 


8 


5 


25 


25 


19 


a 


3 


10 


14 


10 


11 


0 


3 


0 


8 


5 


0 


Special Educator 


7 


8 


? 1 


51 


3 


8 


13 


15 


13 


36 


17 


49 


0 


8 


10 


28 


50 


67 


0 


3 


0 


0 


3 


5 


Vocational Educator 


7 


10 


7 


38 


2 


23 


5 


3 


7 


15 


0 


0 


5 


13 


9 


18 


2 


3 


7 


5 


11 


21 


0 


0 


Supervisor of 
Instruction 


8 


25 


20 


43 


4 


8 


0 


7 


0 


26 


. 12 


74 


8 


14 


8 


28 


12 


13 


8 


7 


4 


8 


8 


28 


Counselor , 
Psychologist 


7 


4 


16 


20 


7 


9 


31 


50 


10 


37 


6 


17 


7 


0 


9 


12 


12 


13 


4 


11 


6 


17 


0 


2 


Researcher 


25 


34 


36 


33 


17 


22 


9 


25 


21 


34 


23 


34 


30 


39 


19 


28 


10 


23 


13 


30 


n 


27 


8 


15 


Instructional 
Resources Specialist 


4 


24 


30 


48 


9 


26 


8 


24 


9 


32 


12 


31 


3 


29 


27 


59 


12 


30 


12 


2f 

l 

| 


: 6 
i_ 


23 


6 


24 


Program Specialist 


H 

30 


_ 

24 


31 


49 


5 


20 


13 


18 


17 


28 


17 


22 


8 


19 


12 


25 


21 


17 


2 


! 

15 


! 

4 


15 


5 


11 


Principal, 

Asst* Principal 


8 


24 


8 


44 


8 


12 


10 


8 


n 


19 


20 


31 


0 


23 


4 


18 


10 


10 


4 


3 


0 


2 


0 


4 


Elementary Teacher 


4 


11 


7 


26 


0 


6 


0 


6 


4 


19 


11 


25 


0 


2 


0 


9 


16 


13 


2 


2 


2 


0 


5 


n 


Secondary Teacher 


2 


6 


9 


24 


1 


2 


0 


2 


3 


2 


0 


1 


1 


2 




10 


1 


2 


1 


2 


0 


1 


4 


25 


College Professor 


6 


11 


21 


43 


4 


15 


4 


12 


7 


21 


10 


21 


7 


17 


n 


22 


11 


11 


11 


18 


7 


14 


6 


13 


College Administrator 


9 


21 


12 


43 


18 


26 


12 


16 


0 


26 


9 


18 


24 


26 


18 


23 


6 


11 


26 


32 


24 


27 


3 


8 


Superintendent , 
Asst* Super. 


44 


47 


i8 


46 


16 


14 


16 


18 


4 


24 


24 


27 


48 


45 


16 


19 


16 


19 


4 


10 


4 


5 


8 


8 


Other Admin. 
Position 


17 


47 


26 


42 


17 


22 

. 


9 


17 


17 


20 


22 


22 


22 


30 


9 


32 


13 


17 


13 


7 


4 


5 


9 


13 
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TABLE V-3 . PERCENTAGES OF SCREENING QUESTIONNAIRE (Ql) RESPONDENTS, BY USER 
GROUP AND SAMPLES, WHO REPORT HAVING READ/SKIMMED PRODUCTS 




0 



3 
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There are several ways to study this table, including tracing across, by user 
gtoup, to the NCEC unit or product that a given group mignt be expected to know 
or by tracing down each unit or product group to note the relative representa- 
tion of different user groups. It is also interesting to contrast the percent- 
ages in this table with those in the "interest" tables (Tables IV-4 and IV-5) 
in the previous chapter. For example, 25 percent of the Secondary Teachers 
(in the non-random sample) expressed "major" interest in languages and linguis- 
tics, and the same percentage reported having read or skimmed products from the 
Languages and Linguistics Clearinghouse. With Superintendents, 80 percent of 
the random sample expressed "major" interest in educational administration, 

but 48 percent reported having read or skimmed products from the Educational 
Management Clearinghouse. 

These percentages are particularly useful in conjunction with those in the 
following table— Table V-4. The combined percentages provide some basis for 
assessing the extent to which groups, particularly those representing intended 
audiences, have been "reached" or "not reached." For example, in the illus- 
tration used earlier with Superintendents, in which 48 percent in the random 
sample had reported reading familiarity with the Educational Management 
Clearinghouse: in Table V-4, 12 percent had "heard about" products from this 
Clearinghouse, and 40 percent in this population of respondents had "no aware- 
ness" of the Clearinghouse's products. Although goals for greater familiarity 
might be established in cases such as this, it should be noted that this per- 
centage of familiarity is probably quite high. In interpreting these general- 
familiarity data, it is important to note that users do not always associate 
reading materials with their originating organizations, such as with ERIC — a fact 
that is well documented in the experiences of local educational information per- 
sonnel who are disseminators of these materials. 
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N I 
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N 


R 


H 


R 


H 


R 


H 


R 


N 


R 


N 


R 


N 


R 


N 


R 


Heading Special isl 


5 


17 


3 


17 


10 


n '> 


20 


17 


15 


14 


5 


14 


10 


22 


10 


17 


10 


14 


5 


b 


5 


8 


0 

17 


Special Educator 


7 


10 


17 


8 


23 


28 


7 


21 


10 


21 


20 


18 


7 


33 


1 * 


28 


17 


10 


17 


21 


10 


18 


Vocational Educator 


18 


13 


34 


21 


23 


28 


9 


26 


16 


18 


14 


23 


14 


15 


14 


18 


11 


23 


1 1 


31 


5 


2 3 


7 


Supervisor of 
Instruction 


20 


9 


28 


17 


16 


26 


16 


20 


28 


18 


16 


21 


8 


26 


12 


17 


i: 


18 


8 


18 


8 


18 


0 


Counselor , 
Psychologist 


9 


9 


16 


18 


12 


22 


15 


17 


6 


16 


16 


26 


7 


21 


; 


16 


12 


28 


; 18 


18 


10 


12 


6 


Researcher 


12 


13 


20 


22 


25 


37 


32 


34 


31 


30 


34 


27 


29 


21 


28 


31 


34 


34 


28 


27 


26 


24 


31 


Instructional 
Resources Specialist 


13 


18 


22 


18 


20 


26 


20 , 


,24 


21 


22 


25 


24 


25 


20 


15 


13 


27 


24 


18 


24 


24 


22 


28 


Program Specialist 


19 


13 


23 


16 


24 


27 


23 


29 


23 


25 


24 


29 


23 


28 


20 


27 


21 


25 


23 


26 


16 


23 


20 


Principal , 

Asst. Principal 


8 


13 


16 


12 


16 


16 


8 


26 


18 


14 


12 


24 


8 


14 


8 


16 


12 


24 


4 


20 


6 


14 


12 


Elementary Teacher 


7 


17 


11 


17 


7 


21 


9 


19 


7 


15 


18 


23 


13 


17 


14 


19 


9 


19 


7 


21 


11 


26 


7 


Secondary Teacher 


3 


4 


7 


20 


5 


9 


8 


12 


5 


9 


7 


8 


7 


7 


11 


10 


7 


9 


5 


9 


5 


10 


9 


College Professor 


14 


14 


19 


15 


24 


25 


24 


23 


20 


22 


24 


21 


17 


19 


21 


20 


19 


23 


23 


23 


24 


18 


14 


College Administ rator 


9 


8 


12 


13 


21 


25 


18 


26 


21 


20 


9 


25 


15 


18 


15 


21 


9 


29 


21 


26 


18 


19 


12 


Supt* intendent , 
Asst. Super. 


12 


8 


24 


23 


16 


21 


24 


20 


20 


18 


28 


20 


y 2 


11 


28 


20 


24 


20 


8 


15 


12 


18 


12 


Other Admin. 
Position 


30 


e 


17 


20 


30 


32 


30 


22 


17 


30 


17 


22 


17 


15 


22 


17 


22 


17 


13 


25 


13 


25 


13 




# 



137 



TABLE V-4 . PERCENTAGES OF SCREENING QUESTIONNAIRE (Ql) RESPONDENTS, BY USER 
GROUP AND SAMPLES, WHO REPORT ONLY HAVING HEARD ABOUT PRODUCTS 





3 
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3 . RE SPONDENTS * FAMILIARITY WITH SPECIFIC PRODUCTS 

Each participant in the Evaluation Questionnaire Survey (Q2) was asked to eval- 
uate 10 products. The level— of— familiarity data, shown in Table V— 5, are based 
on responses to the first questionnaire item: 

» v 

Please look at the document representation on the left and 
indicate your previous awareness of the full document. 

1. ( ) I have not previously seen or read (used) 

this document. 

( ) I have previously seen or heard about this 

document but have not road or skimmed (used) it. 

( ) I have previously read or skimmed (used) this 
document . 



A total of 1,251 respondents contributed 12,453 Reader and Non-reader evaluation* 
responses, presented in Table V-5 by product type and in percentages of "cases/ 
i.e., respondents* evaluations of sets of 10 products. Of these, approximately 
60% contributed at least one Reader evaluation. This percentage representation 
(not the actual sample sizes) is comprise4 equally of random and non-random 
sample participants. 

The "read/ skimmed" group, referred to hereafter as Reader Evaluations, numbers 
1,914 — approximately 15 percent of the total evaluations received. (The percen- 
tage of Specialists with prior familiarity is somewhat higher.) Yet, in 69 per- 
cent of the cases, products had neither been seen nor heard of. 

The findings from further analyses of the familiarity question are presented in 
Tables V-6a, V-6b, and V-6c. Differences are shown in product recognition 
(i.e., reading familiarity) by levels of visibility, levels of effort, and 
subject areas. 



Table V-6a indicates that high-visibility Reviews and Practical Guidance Papers 
are known by a significant (at the .01 level) percentage over low-visibility 



o 

ERIC 



*The total number of evaluations should have been 1,251 time 10, or 12,510. The 
discrepancy is due to the number of unusable returns in respondents* packages 
of 10 questionnaires that could not be included in the evaluation data file 
(discussed in Chapter III) * 



y ■ o 
1 



v-n 



TABLE V-5 . USERS AND SPECIALISTS REPORTING FAMILIARITY 

WITH ASSIGNED DOCUMENTS FROM THE PRODUCT SAMPLE 



A, USERS (from User Evaluation Questionnaire Survey) 

[Respondent N = 1,251] 

[Evaluation N = 12,453] 



Familiarity Level 


BIBLIOGRAPHIES 
[N = 4,133] 


PRACTICAL 
GUIDANCE 
PAPERS 
[N = 2,837] 


REVIEWS 
[V = 5,483] 


Read/ Skimmed (READERS) 


501 (12%) 


426 (16%) 


957 (17%) 


Seen/Heard of (NON-READERS) 


763 (18%) 


419 (15%) 


751 (14%) 


Neither Read/Skimmed 








nor Seen/Heard of (NON-READERS) 


2869 (69%) 


1962 (69%) 


3775 (69%) 



B. SPECIALISTS (from Specialists’ Questionnaire Survey) 







PRACTICAL 








GUIDANCE 






BIBLIOGRAPHIES 


PAPERS 


REVIEWS 


Familiarity Level 


[N = 129] 


[N = 94] 


[N = 208] 


Have Previously 
Read Document 


29 (23%) 


26 (28%) 

j 


52 (26%) 




V-12 




ctf 

sO 

I 

> 

M 

W 

s 




o 

ERIC 



V-13 



IC-i. 



a 

in 

<T> 

CL 



c 

rt 

a 

o 

rt 

PO 

05 

Pi 

CL 



cc 

Pi 

< 

CD 

cn 

CD 

fD 

3 

o 

H 



CO X 



s* - 

H* 



fD 

CL 



O 

H 



a> 

Pi 

h 

P- 

o 

Hh 



X 

Pi 

cn < 
X* fD 
H* 

B ^ 

§ H 

a? m 

CL < 

^ o 
o c 

H 01 

t-» 

a 

CO 

m & 

CL fD 
{ii 
CL 



Hj 



6 



r 1 

w 

< 

w 

r» 

C/1 



*"■4 

o 

a* 



ON 

"-4 



o 

5^ 



4> 



-fs 

2>5 



CO 



O 



VD 

2^ 



*•*4 

>5 



a 

ii 

N> 

M 

CO 

N> 



w 

wi r* 

H* O 

g * 

rt 



a 

it 



w s 

hi m 
ho hi cl 
O H* 
H 
rt 



JN 

CO 



a 

ii 

M 

N> 

o 

LO 



td 
Hi ffi 
Hi h- 
O 0 Q 
M P 4 
rr 



a 

II 

Ln 

4N 

00 

CO 

< 

p 

H 4 

c 

p 

rt 

H* 

O 

D 

CO 



00 



^4 

o 

oN5 



-fs 

5^ 



o> 

H 4 



ON 

5^ 



H 4 

-O 

2^ 



Ln 

$vS 



4> 

>5 



H 4 

hO 

2>S 



hO 

CO 



a 

ii 



w 

Hi 

on Hi 



O 

00 



o 

rt 



a w a 

U Hi CD 
M Hi CL 
CO O H* 
On *i | 
M rt I 



a 

it 

00 

•vj 

CO 



td 
Hi as 

Hi h* 
O OP 
H D 4 
rr 



a 

n 

ro 

* 

00 

CO 

•*4 

w 

< 

p 

M 

C 

P 

rt 

H* 

O 

3 

to 



O 

H 

M 

O 

p 

o 

CJ 

M 

o 



o 

w 



o> 

La 



"4 

H* 

cNS 



CO 

>5 



ro 

o 

5sS 



-P> 

2^ 



« W 

Hi 
ON Hi 
^S 
CO 



o 

H 

rt 



II 



VO 



o 

2^ 



W S 

Hi fD 
Hi CL 
O H» 
ON H C 
4S rt § 



o> 



hO 



CO 

ro 



td 

hi PB 
Hi h- 
O OP 
H P 4 




V-14 




TABLE V-6b. LEVELS OF FAMILIARITY REPORTED BY USER EVALUATION 
QUESTIONNAIRE (Q2) RESPONDENTS FOR EACH PRODUCT 
TYPE BY LEVELS OF EFFORT 



TABLE V-6c. LEVELS OF FAMILIARITY REPORTED BY USER EVALUATION 
QUESTIONNAIRE (Q2) RESPONDENTS FOR EACH PRODUCT 
TYPE BY SUBJECT AREAS 
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products. In Table V-6b, only the high-effort Practical Guidance Papers show 
a significant difference over their low-effort counterparts. The subject area 
analysis did not produce any significant findings. 

The major findings reported here are the high number of cases in which products 
had not previously been read or skimmed and, also, the relatively low number of 
cases in which products had previously been heard about. The findings in 
Chapter VII, from Non-readers and Non-users, are required to pursue the further 
interpretation of these data. At this point, therefore, it is perhaps useful 
to conclude this discussion by providing findings of a descriptive nature 
regarding NCEC announcement and distribution procedures. 



c. 



CURRENT ANNOUNCEMENT AND DISSEMINATION PROCEDURES 



Information presented in this section was obtained from the questionnaires (see 
Appendix B) that were used to obtain level-of-ef fort and level-of-visibility 
data from each NCEC unit and from nine states selected to illustrate the 
secondary distribution of PREP reports. Strategies for announcement and 
distribution of products in the sample are presented below for each product 
group: 1) PREP reports, 2) EMC bibliographies, and 3) ERIC clearinghouse 

products. 

1. PREP REPORTS 

The announcement and dissemination of PREP reports are largely the responsibility 
of state education agencies. USOE-prepared materials — announcement flyers, 
called PREP Briefs, and copies of PREP reports — are distributed to the agencies, 
which are presumably in a better position to identify and respond directly to 
individuals who are in the targeted audiences of the reports. The degree of 
dissemination activity within the states is known to vary widely, from no effort 
at all to fairly large-scale announcement, distribution, and repackaging efforts. 
A survey of these state activities was beyond the scope of this project. How- 
ever, with the aid of USOE, nine states (the maximum number in any population 
that can be surveyed without prior approval of the Office of Management and 
Budget) were selected for study. These states were believed to represent low, 
medium, and high levels of, PREP report dissemination activity. 

A special version of the questionnaire u^ed with NCEC units was designed to 
obtain this information from the states for each of the PREP reports in the 
product sample. With the exception of one state of the seven responding, no 
differences in the handling of PREP reports (by topic) were reported; selected 
results from the seven states displayed in Table V-7 are fairly typical of the 
procedures used with all PREP reports. 
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TABLE V- 7. RESPONSES FROM SELECTED STATES TO QUESTIONS 

ON PREP REPORT DISSEMINATION ACTIVITIES (SHEET 1) 







A. ANNOUNCEMENT 


1. Did you 


distribute 


the Brief for these PREP reports? 




Yes 6 


No 1* 


If yes, please give an estimate of the total number of copies distributed 
and identify the targeted professional groups who were sent copies. 


State 


Copies 


Audiences 


A 


0 




B 


100 


PREP coordinators in school systems throughout the state 


C 


1300 


Regional centers throughout the state 


D 


195 


Regional centers, county school systems, and specialists 


E 


200 


College libraries 


F 


5300 


Regional centers (repackaged excerpts are also mailed to 
specialists) 


G 


900 


Chief school administrators, teacher education institu- 
tions, and teachers 


2. Did you prepare any special announcement materials concerning the avail- 
ability of these PREP reports? 




Yes 4 


No 3 


If yes, 


identify the method of distribution. 


State 


Method 


B 




Journal articles 


C 




Newsletter articles 


D 




Newsletter articles 


E 




Letters and newsletter articles 


* 

One state : 
activities 


reported only recently initiating any PREP report dissemination 
• 
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TABLE V-7 . RESPONSES FROM SELECTED STATES TO QUESTIONS 

ON PREP REPORT DISSEMINATION ACTIVITIES (SHEET 2) 



B. DISTRIBUTION 

■« — W 

. How do you make copies of the PREP reports available to educators in your State? 

1 Copies are reproduced on request 

3 Copies are reproduced and automatically distributed to a selected 

audience 

1 Copies are reproduced and automatically distributed to a selected 
audience, and copies from USOE are made available on- loan 

1 Copies are reproduced and automatically distributed 

1 Copies are reproduced on request; reproduced and automatically 

distributed; circulated; and USOE copies are made available on-loan, 
as requested 



2. As of this date, please provide an estimate of the total number of copies dis- 
tributed by your State for [each] PREP report. 



State 

A 

B 

C 

V 

D 

E 

F 

G 



Number of Copies 



0 

100 

1300 

26 

30 

812 

1600 



(av.) 



(av.) 



3. In recent months, have you changed the procedures described above in any way 
due to the availability of PREP’s from GPO? 

Yes 3 No 3 No Response 1 



State B. "Encourage direct purchase from GPO in order to expedite dissemina- 
tion and to enable individual teachers to procure such materials 
where local administrators are dragging their feet." 

State C. "Less distribution . . .lack of local funds to reproduce. . .encourage 
schools to subscribe." 

State E. "Now distribute our five copies to members of State Department of 
Education; do not reproduce any copies to distribute. Still send 
Briefs to regular mailing list." 
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EMC BIBLIOGRAPHIES 



These documents are' made available through two channels. They are sold through 
the U.S. Government j Printing Office and distributed free by USOE. The total 
number of copies distributed through both channels averages around 8,900 for 
each of the four documents. Other than occasional references in professional 
journals, these bibliographies reportedly received no special publicity. 

3. ERIC CLEARINGHOUSE PRODUCTS 

Clearinghouses supplied information on the types of announcement and distribu- 
tion methods used with their documents in the product sample. Their responses 
to the structured question are shown in Table V-8. 

Other publicity media are used by the clearinghouses; they include mailers to 
special audiences; announcements, advertisements, and exhibits at conventions; 
and press releases. It was reported that some documents received special 
br ibut ion by inclusion in other publications (e.g«, books of readings or 
reviews) by other agencies, such as state education agencies, USOE, or pro- 
fessional associations, and at conventions. Special publicity also resulted 
from the use of documents in seminars or courses as resources by national 
commissions or task forces or from presentation as papers at conventions. A 
few documents received special publicity as award winners, and one was the 
subject of a Voice of America broadcast. 

Clearinghouses also indicated the kinds of distribution channels used for these 
documents, presumably typical for most of their products . The majority of the 
133 ERIC documents (57 percent) were distributed solely by the clearinghouses 
(and then, of course, through tne ERIC Document Reproduction Service). Only 
six documents were distributed solely by another agency, e.g., a professional 
association or university; the remainder were distributed jointly by clearing- 
houses and other agencies. Most documents were available at no cost; however, 
some were sold through other agencies, but were available free through the 
clearinghouses . 
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TABLE V-8. SPECIAL PUBLICITY RECEIVED BY ERIC CLEARINGHOUSE PRODUCTS 



Question: Did this product receive special publicity, other than the usual 

mention in clearinghouse newsletter or association newsletter? 
That is, was special attention drawn to the product in any of 
the following ways? 



r 


ERIC 


CLEARINGHOUSE PRODUCTS 




Reviews 

(N=67) 


Practical 
Guidance Papers 
(N*23) 


Bibli- 

ographies 

(N=43) 


It was reviewed in the professional 
literature. 


31% 


26% 


10% 


It was extensively described — featured — 
in a column, brochure, or other 
publicity medium. 


43% 


40% 


35% 


It received special attention because it 
was part of a larger effort, such as a 
well-publicized product series or a 
professional gathering, such as a 
workshop. 


45% 


35% 


51% 


It came to the attention of a large 
relevant audience because it was 
distributed through a well-established 
channel such as a professional journal. 


12% 


9% 


28% 


It was placed on automatic distribution 
to individuals on our mailing list. 


70% 


48% 


42% 


(Received no special publicity) 


0% 


17% 


0% 



Some limited attempt was made to uncover reviews of products in the professional 
literature. The purpose of this search was to compare such reviews with evalua- 
tions obtained from Specialists; however, the findings in this exercise are 
useful only in reporting the apparent lack of professional reviews in the 
literature. From citations provided by the clearinghouses, we attempted to 
find reviews of 15 Reviews (out of the approximately 20 that were checked). 

Many of these reviews had multiple citations, but the journals, in many cases, 
were too obscure to be available even in the library o.t the National Educational 
Association in Washington, D.C. Reviews of only six of the 15 could be found. 
All but three of these were announcements of the publications. Only one could 
be considered an "outsider’s" critical review; the remainder were summaries of 
the products presented in ERIC columns in professional journals. 

There is a wide variety in the character of the activities within and between 
clearinghouses. A study of the benefits of these activities was not within the 
scope of this project. However, there are indicators of the complexities 
involved in determining the most effective methods for increasing awareness of 
products. Bibliographies from one clearinghouse, that were available through 
professional journals (in ERIC columns) and as reprints, were not known by this 
survey respondent population; and yet, an annually compiled bibliography from 
another clearinghouse, that also appears in a major professional periodical, 
was known. Among the Reviews that were made available in the professional 
journal literature, four out of six had been read by a relatively large number 
of respondents. The need for assessing the effectiveness of these various 
distribution channels is clearly indicated. The different. among user groups, 
in their traditions of usage of the professional journal literature and in the 
strength of their major professional associations as avenues for publicity and 
distribution, may suggest the need for several approaches to increase awareness 
of these products among educators. 
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VI. COMPARISON OF THE TWO SAMPLES IN THE 



GENERAL FIELD SURVEY 



The dual sampling strategy in the General Field Survey was developed to 
accomplish two major study objectives. The random sample* was designed 
particularly to assess the outreach of products, and the non-random sample 
was used to help ensure the inclusion of likely product users — and therefore, 
potential product evaluators. 

The use of these two samples provides some general information concerning the 
"potential " of either or both sampling procedures for identifying product 
evaluators in future evaluation studies. Some comparisons, on background 
variables and levels of familiarity have been discussed in the two preceding 
chapters. Although the data presented in these earlier chapters suggest that 
the two groups are reasonably comparable — and more so than might have been 
expected — there was a need to treat this comparison in a statistical fashion. 
A decision on whether to combine the evaluation data from both samples had to 
be made prior to the performance of the planned analysis . There were also 
practical considerations to be accommodated in this decision. The imbalance 
in size of the two sample groups represented a problem in separately reporting, 
with any degree of confidence, the evaluation data for the smaller random 
sample. Therefore, it was evident that a decision to combine the samples 
would strengthen the overall reporting of the findings. With this tentative 
decision in mind, we decided to pursue the plan for conducting a "convergence 
test" that, if positive, would strengthen the practical decision, and, if 
negative, could be reported for its impact on any subsequent interpretation. 



*The stratified random sample included personnel from state education 
agencies, local school districts, and institutions of higher education 
in eight states (selected randomly from each of the major regions of 
the country); in addition, researchers were sampled from USOE- 
supported research facilities. The non-random sample included names 
sampled from available listings from NCEC-related units, including 
ERIC clearinghouse mailing lists, state and local educational 
information centers, and the ERIC Document Reproduction Service. 
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A profile correlation method, in the tradition of Raymond Cattell’s q correlations 
and the Campbell-Fiske multitrait-multimethod matrix, was originally planned 
for this analysis. However, the small number of random sample respondents — mostly 
heterogeneous groups for given documents — made it unlikely that evaluations of the 
same document would agree substantially between the random and non-random samples. 



Therefore, we decided in favor of a simple comparative display of the overall 
evaluations (of substantive papers) the elements of which follow: 

• Samples . Since there were 1229 non-random-sample evaluations of 
Reviews and Practical Guidance Papers, but only 191 random-sample 
evaluations, it was possible to divide the non-random sample into 
five random subsamples with an average of 246 evaluations each.* 

(It may be confusing td speak of random subsamples from a non-random 
sample; however, what is important is that each of these subsamples 
is still a non-random sample of the population. 

• Display . Table VI-1 contains the computations for the overall 
evaluation of Reviews and Practical Guidance Papers (drawn from 
the same survey instruments) , such that percentages of responses** 
could be displayed separately for each subsample of the non-random 
sample, as well as for the random sample. Each questionnaire item 
(questions 5 through 13 of Q2-A) and each response choice is dis- 
played with the six corresponding percentages. (The last column 
is the average for the total .) 

• Hypothesis . It was hypothesized that the average percentage agree- 
ment among subsamples of the non-random sample would be no greater 
than percentage agreement of the random sample with them. In other 
words, percentages from the random sample would agree as well with 
percentages from various subsamples as the subsamples agreed with 
each other. 



It can be seen in Table VI-1 that the random sample percentages are indis- 
tinguishable from percentages of the five subsamples. No pattern of deviation 
sets the random sample apart. The hypothesis can also he addressed statistically 
by testing for the significance of difference between the random sample and 



*These random subsamples were created by using the last two digits in 
the SDC- assigned document code number. Since these numbers were 
assigned sequentially for the product universe of each NCEC unit, and 
in no special order, there is little likelihood that any biases were 
built into these groupings. 

**Percentages , ere selected over mean scores to provide more data points 
for this comparison. 
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TABLE VI-1. 



COMPARISON OF RANDOM AND NON-RANDOM SAMPLES ON READER 
RESPONSES TO EVALUATION QUESTIONS FOR ALL REVIEWS AND 
PRACTICAL GUIDANCE PAPERS. (SHEET 1) 





NON-RANDOM SAMPLE (RANDOM SUBSAMPLES) 
[Av. N-246] 


RANDOM 

SAMPLE 

[N-191] 


AVERAGE 

[N-1420] 


Reading Recency-1 


11X 


18X 


10X 


7 % 


18X 
1 £ 


12X 

20 


(12X) 
(18 ) 


Reading Recency-2 
Reading Recency-3 
Reading Recency-4 


17 

19 

53 


14 

21 

46 


16 

30 

45 


23 

27 

42 


10 

25 

41 


21 

47 


(24 ) 
(46 ) 


Relevance-1 

Relevance-2 

Relevance-3 


1 

30 

69 


0 

29 

71 


1 

31 

68 


1 

21 

78 


2 

26 

72 


1 

28 

71 


( 1 ) 
(27 ) 
(72 ) 


Need-1 

Need-2 


5 

58 


6 

51 


5 

57 


6 

49 


9 

49 


7 

59 


( 6 ) 
(54 ) 


Need-3 


36 


43 


38 


45 


42 


35 


(40 ) 


Coverage-1 

Coverage-2 

Coverage-3 


3 

48 

48 


3 

46 

50 


2 

49 

49 


2 

45 

53 


2 

40 

58 


3 

52 

45 


( 3 ) 
(47 ) 
(51 ) 


Up-To-Dateness-1 


5 


5 


3 


5 


3 


4 


( 4 ) 


Up-To-Dateness-2 


13 


12 


11 


8 


8 


19 


(12 ) 


Up-To-Dateness-3 


82 


83 


86 


86 


89* 


76 


(84 ) 


Length-0 
Length- 1 
Length-2 
Length- 3 


5 

3 

12 

79 


3 

3 

9 

84 


3 

5 

9 

83 


3 

6 

10 

81 


2 

2 

8 

87 


5 

4 

7 

84 


( 4 ) 
( 4 ) 
(10 ) 
(83 ) 


Organization-1 


3 


2 


3 


2 


1 


3 


( 2 ) 


Organization-2 


60 


64 


64 


63 


61 


69 


(63 ) 


Organization-3 


37 


34 


32 


35 


38 


28 


(34 ) 


Writing Style-1 


3 


3 


2 


2 


3 


3 


( 3 ) 


Writing Style-2 


40 


41 


47 


46 


35* 


50 


(43 ) 


Writing Style-3 


57 


56 


51 


53 


62* 


47 


(54 ) 


Format-1 


2 


2 


1 


0 


0 


0 


( 1 ) 


Format-2 


23 


25 


26 


27 


24 


28 


(25 ) 


Forma t-3 


75 


73 


73 


73 


75 


72 


(74 ) 


Thoughtfulness-1 


5 


5 


4 


4 


s 


6 


( 5 ) 


Thoughtfulness-2 


57 


56 


62 


61 


57 


67 


(60 ) 


Though t f ulnes s - 3 


38 - 


39* 


34 


35 


38 


27 


(36 ) 


Obtain Overview-1 


2 


0 


1 


0 


1 


1 


( 1 ) 


Obtain Overview-2 


43 


40 


34 


36 


36 


41 


(38 ) 


Obtain Overview-3 


56 


59 


66 


64 


63 


58 


(61 ) 


Look Up Facts-1 


4 


5 


4 


6 


6 


6 


( 5 ) 


Look Up Facts-2 


72 


68 


74 


69 


63 


65 


(69 ) 


Look Up Facts-3 


24 


27 


22 


26 


31 


29 


(26 ) 
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TABLE VI-1 



COMPARISON OF RANDOM AND NON-RANDOM SAMPLES ON READER 
RESPONSES TO EVALUATION QUESTIONS FOR ALL REVIEWS AND 
PRACTICAL GUIDANCE PAPERS. (SHEET 2) 





NON- 


-RANDOM 


SAMPLE (RANDOM SUBSAMPLES) 
[Av. N-2463 


RANDOM 

SAMPLE 

[N-191] 


AVERAGE 

[N-1420] 


Identify Indlvlduals-1 


52 


4X 


42 


62 


42 


72 


( 52) 


Identify IndlviduAls-2 


82 


74 


80 


75 


75 


80 


(78 ) 


Identify Individuala-3 


13 


22 


16 


19 


20 


14 


(17 ) 


Identify Relevant Lit.-l 


2 


2 


3 


2 


1 


5 


( 2 ) 


Identify Relevant Lit. -2 


65 


59 


67 


63 


54 


65 


(62 ) 


Identify Relevant Lit. -3 


32 


40 


29 


35 


44* 


30 


(35 ) 


Update Knowledge-1 


3 


2 


2 


3 


1 


3 


( 2 ) 


Update Knowledge-2 


53 


51 


50 


50 


46 


49 


(50 ) 


Update Knowledge-3 


44 


47 


48 


47 


52 


48 


(47 ) 


New Knowledge-1 


4 


4 


4 


4 


4 


5 


( 4 ) 


New Knowledge-2 


79 


78 


79 


77 


70 


74 


(76 ) 


New Knowledge-3 


18 


18 


17 


18 


26 


21 


(20 ) 


To Make Decisions-0 


83 


79 


82 


78 


76 


80 


(80 ) 


To Make Decisions-1 


16 


21 


18 


22 


24 


20 


(20 ) 


Applied To Work-0 


38 


30 


29 


33 


29 


37 


(33 ) 


Applied To Work-1 


62 


70 


71 


67 


71 


63 


(67 ) 


To Give Advice-0 


57 


52 


59 


55* 


48- 


62 


(56 ) 


To Give Advice-1 


43 


48 


41 


45 


52* 


38 


(44 ) 


Examined Documente-0 


77 


64 


72 


69 


64 


72 


(70 ) 


Examined Documents-1 


23 


36 


28 


31 


36 


28 


(30 ) 


Consulted Author-0 


92 


91 


94 


93 


94 


91 


(92 ) 


Consulted Author-1 


8 


9 


6 


7 


6 


9 


( 8 ) 


Passed To Colleague-0 


53 


60 


52 


55 


44 


53 


(53 ) 


Passed To Colleague-1 


47 


40 


48 


45 


56 


47 


(47 ) 


Comparative Usefulnesa-1 


2 


2 


2 


1 


1 


2 


( 2 ) 


Comparative Usefulness-2 


43 


42 


44 


37 


35 


42 


(41 ) 


Comparative Usefulnesa-3 


55 


57 


54 


62 


63 


55 


(58 ) 
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each subsample of the non-random sample (i.e., column six against each of the 
first five columns). There were, as indicated by asterisked percentages in 
the table, only eight cases in which the response percentages of the random 
sample differed significantly (at the .01 level) with one non-random subsample 
We do not believe that the significant differences in eight paired comparisons 
less than 2% of the possible 325, provide sufficient evidence to judge the 
samples as coming from different populations. These results are accepted as 
confirming the null hypothesis and supporting our decision to combine the 
evaluation data from both samples in the several analyses discussed in the 
next chapter. 
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EVALUATION DATA 



VII. REPORT ON THE EVALUATION DATA OF 

NCEC INFORMATION ANALYSIS PRODUCTS 



A. INTRODUCTION 

Evaluation data presented in this chapter are based on returns of three survey 
instruments: (1) the User Evaluation Questionnaire (Q2) , (2) the Non-User 

Evaluation Questionnaire (Q3) , and (3) the Specialists’ Questionnaire (Q4) . 
Table VII-1 shows types of responses and evaluations drawn from these three 



sources , 

TABLE VII-1. CLASSES OF RESPONSES AND RESPONDENTS FROM 
EVALUATION SURVEY INSTRUMENTS 



QUESTIONNAIRE 


GROUP TO WHOM 
DIRECTED 


POSSIBLE RESPONSE'' TO 
PARTICULAR DOCUMENTS 


EVALUATOR-GROUP 

DESIGNATION 


Q2 


Reported Users 
of NCEC Products 


Read / skimmed 


’’Reader" 


Did not read or skim 


"Non-reader” 


Q3 


Reported Non- 
users of NCEC 
Products 


Read / skimmed 


"Reader”* 


Did not read or skim 


"Non-user" 


Q4 


Specialists in 
Particular 
Subject Areas 


(Full document 
provided) 


"Specialist" 



*Only 10 respondents to Q3 were Readers rather than Non-users. This group was 
too small to be included in any of the following analyses. 



Hereafter, the data sources will be referred to as Readers, Non-readers, Non-users, 
and Specialists. (See Appendix G for a ’’ready reference” to the definition of terms.) 

The focus in this chapter is on data that have been aggregated by product type, 
and within this basic grouping, by document and user group characteristics. An 
underlying at^umption in these, as in all aggregate analyses, is made on the 
homogeneity of the groupings. (Volume II of this report presents evaluation 
data on each of the 146 documents in the product sample.) In this introduction, 
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the analysis variables and statistical procedures used are discussed and an 
outline of the remainder of the chapter is given. 

1. ANALYSIS VARIABLES 

Five dependent variables were used in this study to create groupings of 
evaluations from which valid generalizations could possibly be drawn. Four of 
these variables — product type, level of effort, level of visibility, and 
subject area — are characteristics of documents; the final one characterizes 
the respondent population by user group. Although the development or derivation 
of most of these variables has been discussed in some detail in previous chapters 
(III and IV), a brief discussion of each is presented below for quick reference. 

Product Type . The basic dimension upon which all data have been aggregated 
is the product-type dimension. All documents in the sample are classified 
as one of three types: 

• Reviews and State-of-the-Art Papers, from ERIC Clearinghouses 
(N-68) 

• Practical Guidance Papers, including ERIC Clearinghouse- 
prepared documents and PREP reports (N«*31) 

• Bibliographies, including ERIC-prepared documents and EMC 
bibliographies (N»47) 

Some consideration was given to the possibility of performing an aggregate 
analysis on all 146 documents. However, because of the differences in purpose, 
use, and intended audiences among the product types, this possibility was not 
pursued. 
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Level-of-Ef fort-index . Level-of-effort indexes were created for each product 
type. These were developed from the distribution of professional man-hours 
invested in the preparation of the documents, data for which were provided by 
each NCEC originating unit. The index for each product type is shown below: 





BIBLIOGRAPHIES 


PRACTICAL GUIDANCE 
PAPERS 


.REVIEWS 


Index 


Hours No, of Docs. 


Hours No. of Docs. 


Hours No. of Docs. 


Low 

Medium 

High 


10-41 22 

50-80 16 

100-1,250 9 


10-50 9 

75-600 - 15 

1,216-3,248 7 


30-160 25 
175-250 17 
275-3,260 16 



Level-of-Visibilltv Index . The degree of initial exposure that a document 
received in the distribution of original copies is represented in a level-of 
visibility index. Indexes for each product type are displayed below: 





BIBLIOGRAPHIES 


PRACTICAL GUIDANCE 
PAPERS 


REVIEWS 

•*** 


Index 


Conies No. of Docs. 


Copies No. of Docs. 


Copies No. of Docs. 


Low 

Medium 

High 


100-950 25 

2,000-10,500 17 

35,310-63,133 7 


15-350 10 

400-1,627 10 

2,000-11,000 11 


50-950 29 
1,000-2,521 25 
3,300-40,000 14 



Subject Area. Preliminary to these analyses, the previously used subject classi- 
fication (that of the ERIC clearinghouse structure) was further classified into 
four broad educational areas. These are: 



Area No. of Docs . 

Instructional Content 49 
Educational Administration/Services 34 
Special and Other Educational Groups 38 
Higher Education 25 



PREP reports are an exception to this criterion of "original distribution" as 
a measure of visibility. Some estimate of secondary distribution through state 
education agencies is represented in their visibility index level. 
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Each document was assigned to only one of the four areas, and documents from 
any given NCEC unit can be found throughout the four areas. (Specific 
assignments are documented in the product sample listing in Appendix A.) 

User Groups. Fifteen general user groups were created from background infor- 
mation provided by respondents in their Screening Questionnaire (Ql) returns. 
The derivation of this typology is discussed fully in Chapter IV; the groups 
are listed below: 



Reading Specialists 
Special Educators 
Vocational Educators 
Supervisors of Instruction 
Counselors, Psychologists 
Researchers 

Instructional Resources Specialists 
Program Specialists 



Principals, Assistant Principals 
in Element ary /Secondary Schools 

Elementary Teachers 

Secondary Teachers 

College Professors 

College Administrators 

Superintendents in Local and State 
Education Agencies 

Other Administrative Positions in 
Local and State Education Agencies 



The sample sizes for these groups and the case size for the previously mentioned 
analyses, are discussed in the next section. 



2. STATISTICAL PROCEDURES 

As indicated in Chapter VI, statistical evidence supported the convergence of 
the two samples from the General Field Survey. Sample sizes created through 
these combined data were sufficient to provide a sound data base for analysis. 

By product type, the sample sizes are: 957 for Reviews, 456 for Practical 

Guidance Papers, and 501 for Bibliographies. In the secondary levels of analysis 
by document and respondent characteristics, the number of cases was not always 
sufficient to support meaningful interpretations. Means and percentages were 
computed on these small sample sizes, but the reader is alerted to the statis- 
tical instability of the results. (The full range of sample sizes for each of 
the various breakdowns is shown in Table VII-58.) In the aggregate, the number 
of Specialists evaluations was also sufficient for the purposes of analysis: 
for Reviews, 208; for Practical Guidance Papers, 94: and for Bibliographies, 129. 
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Two statistics are used in reporting data for the General Field Survey. Means 
were computed on questions for which responses were taken to represent choice 
points on a scale from one to three (low to high). Percentages were generally 
used on binary choice questions and for multiple-response items. Percentages 
were also used in some of the summary and special analyses to distinguish 
clearly between high and low responses. 

Specialists’ data were tallied manually and are reported, in the aggregate, in 
percentages. 

Interpretation of Data . Points on the scale (from 1 to 3) vary for each item 
because of the value judgments expressed in the response choices of the question- 
naire item. In some cases, a 2.0 represents a neutral attitude; in others, it 
is a favorable response in the continuum. Therefore, we encourage readers of 
this report to relate the mean scores to the words of the choice points. For 
this purpose, the original questionnaire item and response choices are provided 
in each of the tables in Sections B, C, and D. 



Means are rounded off at two decimal points to help clarify the overall pattern 
of evaluation clusters in the middle range. However, the differences, 
whether in overall trends or between two specific cases (e.g., between evaluations 
of high-level-of-ef fort documents and low-level-of-ef fort documents) are not always 
obvious from a visual perusal. 



To help the reader in understanding the survey findings, two approaches are used: 

• Elements of each dimension (of quality, utility, and impact) are 
displayed In single tables. In these displays, means and percentages 
are related directly to the original questionnaire item and the 
response ■ ; .oice . 

• In discussing these tables, we also draw upon several secondary 
analyses that were performed by user group and by subject area. 

The data for these discussions are presented in Section I (pages 
VII-96 through 118). For each element (e.g., topic of a 

questionnaire item related to ''quality"), we have looked only at 
those groups on the extremes, to understand better how dif- 
ferent (at the .01 level) their evaluations are from the norm. 
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Attention is drawn to these differences not to ditract from the overall positive 
findings but rather, to note indicators of satisfaction greater than or less 
than the generally expressed level. Moreover, no statements are available on 
acceptable levels of dissatisfaction, nor on tolerance levels for low ends of 
any distribution. These data on groups of users or groups of documents, for 
which the level is significantly different, are useful as starting points for 
examining evaluations of individual documents contained within groupings. 

Complete tables of all quality, utility, and impact data for each analysis are 
grouped at the conclusion of this chapter. The final table is a tool to assist 
the reader who wishes to perform his own calculations on the data in the indi- 
vidual tables. This table lists the sample sizes for Readers by analysis group. 

In addition, it presents rule-of-thumb reference data for determining the signi- 
ficance of differences between two means at both the .01 and .05 levels of 
confidence. Instructions for computing these differences are provided. 

Comparison of Means . The Reference Mean — the grand mean for the overall evalua- 
tions of any one product type — is always the first of two means that are compared. 
If the sample sizes of the second-mean groups were large and, therefore, would 
have contributed heavily to the Reference Means, then the respondents of the 
second-mean groups were deleted from the total population and the Reference Means 
re-computed. If, as was the case with several of the user groups, the sample 
sizes were fairly small — from 10 to 50 — these intermediate computations were not 
performed. (If sample sizes are less than 10, the existence of differences is 
still discusseu, but no reference is made to their statistical significance.) 

3. ORGANIZATION OF CHAPTER 

In Exhibits 1A, IB and 1C, we present document evaluation profiles from Volume II 
to review the types of basic data supporting the aggregate data. The remaining 
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EXHIBIT 1A. COVER PAGE OF INDIVIDUAL DOCUMENT EVALUATION PROFILE FOR 
DOCUMENT 86 (from Volume II) 



Document No. 86 ERIC Products 1969-1970 . 1970. (ED 041 598) 



NCEC Unit: Library and Information Sciences Clearingho use 

Product Type: Bibliography Level of Effort Index: High 

Subject Cluster: Educational Administration and Visibility Index: Medium 

Services 



GENERAL FIELD SURVEY (N« 204) 

FAMILIARITY 

20 X Previously Read/Skimmed 21 * Only Heard About/Seen 59 X Not Seen/Read 

R ECENCY OF READING 
(N“ 41) 

29 X Within past month 29 % Within past 6 months 

12 X Within past 3 months 29 X More than 6 months ago 

COMMENTS 

READERS: Instr. Resources Spec : in my work in State department with ERIC and 

professional library, is useful in summarizing available materials for staff members... 
would be more useful if clearly stated that is a selected list... or is it compre- 
hensive? Instr. Resources Spec : bibs are great if only libraries would stock the 

contents. Other Admin : needs broader coverage. Prog. Spec : great help in program 

planning. . .usually first step is to see what ERIC products are available. . .thanks. 
Instr. Resources Spec : excellent. Instr. Resources Spec : needs cross-indexing 

by subject areas as there is overlap in clearinghouse products. 

NON-READERS: Instr. Resources Spec : just came to my attention. Sec. Teacher : need 

better distribution and availability to the classroom teacher. Instr. Resources 
Spec : faculty will not be bothered to drive 28 miles to center where indexes are 
available along with the microfiche. 

SPECIALISTS 1 SURVEY (N^3) 

RECENCY OF READING 
(N-O) 

Within past month Within past 6 months 

Within past 3 months More than 6 months ago 

Cannot recall 

COMMENTS 

• Authors Included are good. . .absence of certain authors regrettable. Am surprised 
that articles from publisher periodicals, e.g., "Adult Leadership," appear here. 

• Annotations succinct, informative, well phrased. Since this is an annual pub- 
lication, it serves as a dependable compilation of existing material and becomes 
more useful as issues cumulate. 

• Annotations particularly helpful. Document useful to supplement such sources 
as Education Index and Library Literature. 
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EXHIBIT IB. READER AND NON-READER DATA FOR DOCUMENT 86 (from Volume II) 



(Document 86_ continued) 



READER EVALUATIONS (N-41) 



Mean 



QUALITY 

Reference Mean 



Percentage 



Reference 

Percentage 



Coverage 


2.56 


(2.49) 


No. of references: 




Up-to-dateness 


2.83 


(2.78) 


About right 


83% 


Organiration 


2.17 


(2.23) 


Too many 


_7% 


Format 


2.66 


(2.72) 


Too few 


_ 7% 


Textual material 


2.49 


(2.47) 







(81%) 

(_4%) 

( 11 %) 





UTILITY 








Mean Reference Mean 






Relevance 2.73 


(2.77) 






Need 2.37 


(2.39) 






nntnparative usefulness 2.61 


(2.70) 


Reference 


Purpose 


of use: 


Percentage 


Percentage 


To 


identify documents on particular topics 


7_3% 


(73%) 


To 


identify documents on particular projects 


24% 


(41%) 


To 


identify documents by particular individuals 


10% 


(13%) 


To 


identify documents from particular institutions 


12% 


(11%) 


To 


perform comprehensive search of literature 


51% 


(55%) 


To 


nee kinds of new work being reported 


61% 


(67%) 




IMPACT 






Were cited documents examined? Yes_33_ (80%) Was content of cited 

document (s) as expected 
from bibliographic 
reference? Yes 78 


% No 22 % 


r 

./ 

4* 



NON-READER EVALUATIONS (N- 120) 
Utility 






Relevance 

Potential 

usefulness 



High 


Medium 


Low 


41% 


39% 


19% 


25% 


41% 


32% 



Reasons for not reading : 
~ (N»43) 



21 % Could not readily obtain a copy 



23 X Not sufficiently interested 
9 % Lack of time 



35 % Other 
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EXHIBIT 1C. SPECIALISTS * DATA FOR DOCUMENT 86 (from Volume II) 



(Document 86 continued) 

SPECIALISTS* EVALUATIONS (N-3) 



QUALITY 





Excellent 


Good 


Fair 


Poor 


Not 

Applicable 


No 

Response 


Choice of author 




1 






2 




Selection of 

content/aaterial 




2 






1 




Choice of references 




2 










Inclusion of current 
material 


1 


2 






* 




Accuracy 


1 


1 






1 




Interpretation 




1 






2 




Organisation 


1 








2 




Organisation of 
references 


1 


1 


1 








Format 




2 


1 








Writing 




2 






1 





UTILITY 



Would you recommend to colleagues? 
Yes 3 No 



Usefulness for Various Purposes 



If yes: 

Purpose of Use 


Very 

Useful 


Somewhat 

Useful 


Not At 
All Useful 


No 

Response 


Obtain overview 


1 






2 


Look up facts 




1 


1 


1 


Identify relevant literature 


3 








Identify individuals or institutions 




3 






Update knowledge 


2 


1 






Obtain new knowledge 


2 


1 






Obtain practical guidance 




3 






Other : 










i 









Need for Document of This Type 

1 Very great 

2 Moderately great 

Not at all great 



Overall Usefulness of Document 
_2 It is a very useful document. 



It is not unusually useful, but 
it is worth having available. 

Its usefulness is too limited 
to justify its publication. 



tables are presented in an order that parallels the major issues addressed 



in this study: 




• Section B: 


Overall Evaluations of Each Product Type, by Each 
Responden Group and by Dimensions of Quality. 


• Section C: 


Overall Evaluations of Each Product Type, by Each 
Respondent Group and by Dimensions of Utility. 


• Section D: 


Overall Evaluations of Each Product Type, by 
Readers and by the Dimension of Impact. 


• Section E: 


Evaluations by Product Type and Levels of Effort. 


• Section F: 


Comparison of Evaluations by Intended Audiences. 


• Section G: 


Evaluation of Products in the "Disadvantaged” Subject 
Area. 


• Section H: 


Evaluation by Product Type and Levels of Visibility. 



Finally, Section I presents summary reference tables on selected survey variables 



As available, data from one or more of the survey instruments are brought 
together to contribute to any question that is being discussed. However, most 
data are drawn from the Reader (Q2) evaluations in the General Field Survey. 



B. EVALUATIONS ON THE DIMENSIONS OF QUALITY 



Each of the survey instruments included several items intended to elicit eval 
uator judgments on product quality. The elements of quality were listed earlier 

in Table III- 10 and are described in this section as follows: 

1. Treatment of Subject 

• Selection of Content /Material and Coverage 

• Length of Document and Number of References 

• Choice of References 

• Discussion and Interpretation 

2. Up-to-Dateness 

3. Organization and Clarity of Writing 

4. Format 

5. Choice of Author and Accuracy in Reporting 

As noted earlier, the General Field Survey participants (Q2) were asked to 
judge items relative to their personal needs, i.e., how adequate the quality 
of the product was for their own purposes. Although comparable items were 
included in Q4, the Specialists’ framework for evaluation was intentionally 
related to the needs of professionals in the field, in general, rather than 
personal needs. Therefore, the Specialists’ quality ratings and comments are 
assumed to have been made on the basis of their understanding of the knowledge 
base of literature and practice. For reporting purposes, data from both sources- 
Readers from Q2 and Specialists from Q4— are displayed in the same table in those 
instances where questionnaire items cover the same, or related areas of quality. 

This section also draws upon the collection of tables at the end of the chapter, 
particularly results of analyses by subject area and user group. 

Examples of comments from both survey groups are introduced to clarify the 
respondents r interpretations of the questions, to support the data, or to 
balance the positive ratings with examples of expressions of concern. 



1 . 



TREATMENT OF SUBJECT 



Four areas of quality are discussed: (1) selection of content and coverage, 

(2) length, (3) choice of references, and (4) discussion and interpretation. 
Examples of respondents' comments are presented in the final paragraphs. 

Selection of Content and Coverage of Topic . As pointed out later in Section C, 
Readers tended to use substantive products — Reviews and Practical Guidance 
Papers — primarily to obtain overviews and secondarily to update knowledge 
about subjects already known to them. In this context of need, Readers reported, 
as shown in Table VII-2, that the documents covered the topics somewhat better 
than "moderately well." There is a significant difference (at or beyond the 
.05 level of confidence) between the satisfaction reported for Reviews (2.50) 
and that for Practical Guidance Papers (2.43). 

A 

W 

Several user groups deviated from this expression of general satisfaction. 

(Except for the less-than-10 sample size groups, these differences are signif- 
icant at the .01 level.) For example, Elementary Teachers and Reading 
Specialists (both small samples) rated the coverage of Reviews lower (2.33) 

than the Reference Mean , but Secondary Teachers rated coverage significantly 
higher (2.81) than the norm. 

Elementary Teachers uniformly reported that the topics in Practical Guidance 
Papers were covered very well (3.00). As might be expected, for this partic- 
ular product type, Researchers expressed less satisfaction (2.25). Counselors 
represented the least satisfied group (2.24). 

The small sample-size groups were again on the extremes for bibliographies. 
Reading Specialists were low, with 2.30, and Elementary Teachers, with a mean 
of 2.60, were considerably higher than the Reference Mean in their ratings of 
coverage . 



* 

Reference means are the means of the overall evaluations for a given product- 
type group. They are norms used throughout this chapter for comparing product 
types and analysis groups within product types. 
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TABLE VII-2. READERS’ AND SPECIALISTS’ EVALUATIONS ON "COVERAGE" AND 

"SELECTION OF CONTENT /MATERIAL” DIMENSIONS FOR ALL 
PRODUCT TYPES 
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Although not significantly different from the norm, a pattern of consistently 

lower ratings is evident for documents across all product types in the area of 
Higher Education. No significant differences occur in the general analysis oy 
subject area, which is possibly due to the way in which documents were assigned 
to each area. The intended audiences of each area comprise a heterogeneous 
group of users, and the distribution of their evaluations throughout the docu- 
ments most likely reconciles the specific user-group differences noted above. 

Table VII-2 shows the Specialists' evaluations on the selection of content/ 
material for discussion . As illustrated later in the Commons section, the 
specialists interpreted the questionnaire item basically in terms of coverage, 
which they rated as good/excellent in over 50 percent of the cases. Their 
ratings were slightly lower for Bibliographies than for the other two product 
types. 

Length . The question concerning length was posed, differently for substantive 
papers than for bibliographies. For Reviews and Practical Guidance Papers, the 
question was one of length of the document; for Bibliographies, it was on the 
number of references. The choice points were the same, and data are displayed 
together in Table VII-3. The results in both cases are impressively positive. 

There are some slight differences among different user groups, one of the most 
interesting of which is the second-choice rating by Elementary Teachers. For 
both Reviews and Practical Guidance papers, respondents (again a small group) 
contrasted with most of the other groups in their second-choice response by 
being on the "too long" side of the issue; the second highest percentages for 
most of the other groups were on the "too short" side. 



The letter "R" stands for Reviews; the letter "P M , for Practical Guidance; 
and the letter "B", for Bibliographies. The R-P-B order is used throughout 
the report. The percentages for any given type of product will not always 
total 100, because the "Not Applicable" responses and non-responses are not 
shown in the tables. (Complete data are provided in Section I.) 



o 

ERIC 
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TABLE VII-3 . READERS’ EVALUATIONS ON DIMENSIONS OF "LENGTH” FOR REVIEWS 

AND PRACTICAL GUIDANCE PAPERS AND "NUMBER OF REFERENCES” 
FOR BIBLIOGRAPHIES 




I 



9l Reference s. Only data for Bibliographies are displayed in Table VII- 
They Indicate that most of the ratings (60 percent) were in the good/excellent 
range, with only 20 percent in the poor/fair categories. (Distributions of non 
applicable or no-response percentages are shown in Tables VII-55, 56, and 57. 



TABLE VII-4. SPECIALISTS’ EVALUATIONS ON "CHOICE OF REFERENCES" DIMENSION 
FOR BIBLIOGRAPHIES 



SPECIALISTS* EVALUATIONS 

Choice of References in reference list for B- biographies (question 5c of 




For reference listings supporting substantive papers, the Specialists’ ratings 
(see Tables VII-55 and 56) were ever; more favorable, with 84 percent in the 

good /excellent range for Reviews and 65 percent, for Practical Guidance Papers. 

f 

As shown later by the types of comments obtained from the Specialists, concern 
is often expressed for the omission of references important to the evaluator 
or for the general incompleteness of the listings. Several comments were made 
about the fact that only documents in the ERIC system were cited, particularly 
when authors or compilers themselves made frequent reference to this fact, as 
though it were a limitation. 



Discussion and Interpretation . General Survey participants were asked to evaluate 
the adequacy of the discussion of Reviews and Practical Guidance Papers for their 
purposes, and the usefulness of textual materials (annotations, abstracts, summar- 
ies, etc.) in Bibliographies. Choice points on the scales were different, as 
pointed out in Table VII-5. 

TABLE VII-5. READERS’ EVALUATIONS ON DIMENSIONS OF "DISCUSSION" FOR REVIEWS AND 
PRACTICAL GUIDANCE PAPERS AND "USEFULNESS OF TEXTUAL MATERIALS" IN 
BIBLIOGRAPHIES 



READERS' EVALUATIONS 



Discussion (question lOe of Q2-A) . The discussion was: 

Textual Materials (question lOd of Q2-B) . The textual material (annotations, 
abstracts, summaries, etc.) was: 



1 I I I 1 I I I 

1.0 1.5 

Q2-A: Inadequate 

for my 
purposes 

Q2-B : Inadequate 

for my 
purposes 



P (2.30) 
R (2.32) 
B (2.49) 



2.0 2.5 3.0 

Reasonably Very 

thoughtful thoughtful 

Moderately Very 

useful useful 
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The overall means for the two types of substantive papers were 2.32 for Reviews 
and 2.30 for Practical Guidance Papers. Secondary Teachers were significantly 
above the Reference Mean for Reviews, with 2.65, and Elementary Teachers were 
on the opposite end, with 2.11. Special Educators, with 2.17, were significantly 
below the norm for Reviews. 

For Practical Guidance Papers, Elementary Teachers, with a mean of 2.00, were 
again on the low end of the distribution in evaluating the thoughtfulness of 
the discussions. Researchers, with 2.14, were also lower than the norm, and 
significantly so. 

The ratings for the usefulness of textual materials in Bibliographies indicate 
that these materials are useful. The importance of these bibliographic aids 
is underscored in many of the comments obtained from both survey groups. 

A more in-depth view of the discussion/interpretation variable was obtained from 
Specialists. The results of this item on interpretation, defined as thought- 
fulness, clarity, defensibility , and/or originality in drawing recommendations 
and conclusions, as shown in Table VII-6,*are positive. 

The interpretation item was seen by the Specialist group as most applicable to 
Reviews and Practical Guidance Papers. In 40 percent of the cases with Biblio- 
graphies, evaluators either did not feel that the item was applicable or simply 
did not respond; but, of course, not all bibliographies included discussions. 
Bibliographies that drew the most comments related to interpretation were usually 
those that included brief, introductory analyses or summaries. 

Comments . For each product type, examples of pertinent comments are drawn from 
both survey groups. The numbers in parentheses at the end of the comments are 
references to the full bibliographic citations provided in Appendix A of this 
report. These comments are drawn out of context of the total evaluation of the 
document presented in Volume II and are provided only to illustrate interpreta- 
tions of the questions and to show, within the context of overall favorable 
findings, some constructive criticisms for consideration in the preparation of 
future products. 
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TABLE VII -6. SPECIALISTS' EVALUATIONS ON "INTERPRETATION” DIMENSION BY 
PRODUCT TYPE 
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For Reviews: 



• Limitation of sources to ERIC documents is recognized. (15) 

• Does not clearly identify the most significant gaps in knowledge 
or practice and thus no desired leadership to future research or 
practice. (16) 

• Given limited literature available, it was excellent .. .apparent 
inconsistency [in my responses] because had a great need for 
comprehensive study, and while document did not measure up, 
better than anything else available. (19) 

• Useful for bringing together body of survey research material but. . 
no critical analysis .. .non-critical surveys essentially mindless 
exercises. (20) 

• Interpretation sound though speculative. (41) 

• How appropriate for ERIC editor to prepare lead article, 
plus select reinforcing papers? (35) 

• Length precludes in-depth discussion. . .as starting point, 
high marks; as an analysis, has some shortcomings. (49) 

t A very fine paper that exhausts the subject, so far as I am 
aware. (55E) 

• A contemporary document without some acknowledgement of 
previous considerations. (65) 

• More practical information needed about implementation. (109) 

• Clear but not thoughtfully defensible, accoraing to my under- 
standing of whole topic of differentiated staffing. (120) 

For Practical Guidance Papers : 

• Presents only one side of issue... view traditional and 
presents little new. (18) 

• Programs thoughtfully and clearly explained .. .many new 
Federal programs not included. (4) 

• Would like to have known dates of implementation of each 
project. So brief, it is most useful as a point of reference 
only, or for identification purposes. Interested in more 
information such as feasibility of projects. (5) 




VII-20 

‘ ' 1 

JL - • 



• A good springboard document that could get someone started 
on further reading. (27) 

• A useful piece, somewhat lacking in musical sophistication, 
especially with regard to pitch.... (80) 

• Biases of committee clear and well-stated; thus, recommendations 
easier to interpret. (133) 

For Bibliographies ; 

• Material very incomplete; many worthwhile texts not included. (8) 

• A bibliography of detailed bibliographies in these fields, 
containing reliable analytical reviews, would be more 
worthwhile. (9) 

• Good on studies of schools done fairly recently. Short on 
classics and on non-education sources that could have 
theoretical application. (43) 

• Comprehensive coverage. (79) 

• Annotations succinct and informative. . .Particularly helpful. (86) 

• Interpretation biggest weakness. . .some would have made 
it more helpful. (91) 

• What was criterion for accepting/rejecting item in bibliography? (91) 

• Most annotations short, but give reader clue as to what he will 
find in document. However, several merely give ERIC descriptors, 
which are not sufficient. (114) 

• May need to cite items from the ultra-conservative side, for 
there are needed ideas from that side which teachers should 
be aware of. (122) 

• This summary would give an inexperienced, beginning, or 
curious teacher a start. (128) 

• Document has two main strengths: selection of references 

excellent; annotation thorough.... (130) 

Some of these comments and criticisms on the three types of products are 
directed at selection; others at treatment. There are probably some biases of 
the evaluators in operation, but there are also clear indications that the 
criteria for selection and scope of coverage need to be stated more clearly in 
the documents. The document (or perhaps the descriptor system) should also 
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indicate the rationale for particular levels of treatment, sc that the document 
can be selected by the user and judged accordingly. Also, as suggested earlier, 
there is a need for textual materials (summaries and annotations) to accompany 
bibliographies. This conclusion is supported by the positive comments about 
those that do include them. 

2. UP-TO-DATENESS 

Up-to-dateness is difficult to assess. Publication lag time is a well-known 
problem in professional communication, but the information analysis product 
suffers another, even more serious problem; by its very nature, it requires 
existing literature from which to draw syntheses or bibliographic listings. 

Thus, a new source of delay is introduced into the publication cycle. A second 
problem in timelieness is that the longevity of product usefulness varies consid- 
erably; this introduces additional considerations, such as the rate of change in 
a field, and new versus old ideas. In deference to these complexities, and to 
the need for a fair assessment for the products in the sample, SDC’s approach 
in phrasing the questionnaire item was to ask the evaluators to consider the 
up-to-dateness of each product at the time of its publication . This exacted a 
burden on the reader to step back in time and memory, but hopefully it pre- 
cluded any automatic reactions of obsolescence to dates of 1969 and 1970. 

As background data for further interpretation of the results of this question, 
data concerning "recency of reading" for both Specialists and Readers are 
presented in Table VII-7 . (It is important to recall that the Specialists 
received complete copies of documents and were not expected to have read the 
documents prior to participating in the study.) These data suggest that use of 
the documents has been both recent and perhaps wall prior to the study release. 
For both past and more current use, however, the data in Table VII-8 indicate 
that, on the whole, the products were considered up-to-date. Although the 
spread among Specialists, for the different product types, is more striking 
than it is for Readers, 70 percent of the Specialists' evaluations are still 
in the good/excellent range. Perhaps because they had the doc ume nts in hand, 
the Specialists tended to reflect (as shown In their comments) on the obsoles- 
cence of materials, or to suggest the need for immediate or periodic updating 
of certain products. 



TABLE VII-7. 



RECENCY OF READING PRODUCTS REPORTED BY READERS AND SPECIALISTS 



A. READERS 




♦ 






Reviews 
[N=957 ] 


Practical 

Guidance 

Papers 

[N“456] 


Bibliographies 

[N“501] 


Within the past month 


11% 


13% 


14% 


Within the past 3 months 


15% 


21% 


16% 


Within the past 6 months 


24% 


24% 


30% 


More than 6 months ago 


48% 


40% 


38% 


B. SPECIALISTS 


Reviews 

[N=52] 


Practical 

Guidance 

Papers 

[N=26] 


Bibliographies 

[N-29] 


Cannot recall 


10% 


12% 


14% 


Within the past month 


0% 


0% 


7% 


Within the past 3 months 


10% 


4% 


7% 


Within the past 6 months 


17% 


23% 


10% 


More than 6 months ago 


64% 


58% 


62% 



Certain user groups among the General Survey participants might be expected to 
have more stringent requirements for current materials. Two such groups. 
Researchers and Instructional Resources Specialists, were significantly 

above the mean for both Reviews and Bibliographies. For reviews, the 
means were 2.79 and 2,88, respectively; for Bibliographies, 2.72 and 2.78. In 
the analyses by general subject area. Reviews in the area of Special and Other 
Educational Groups, with a mean of 2.74, fell significantly short of the 
Reference Mean. 




t 
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TABLE VII-8 . READERS f AND SPECIALISTS’ EVALUATIONS ON "UP-TO-DATENESS" OF 

ALL PRODUCT TYPES 
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Illustrative comments on the issue of up-to-dateness include: 



For Reviews: 



• Not up to date; did not include my own directly relevant research. (19) 

• Most references old. (20) 

• Same defects as all ERIC reviews I encounter ... long lag time between 
appearance items... too much on what is already common knowledge* (51) 

• Very timely article. Because of recent court rulings about 
termination of non-tenured faculty, recruitment becomes even more 
important, and an article reviewing literature since 1969 is 
needed. (73) 

• Needs updating in 1972 .. .already weak in 1969 on objective 
concerning foreign life-style and literature. (83) 



For Practical Guidance Papers : 

• Too much reliance on outdated, erroneous studies. Some very obsolete 
and inaccurate notions of "language." (1C) 

• Undated. (2) 

• Need current information on projects across country made available 
on frequent basis. (2) 

• [an October 1969 document] One of the most useful documents ever 
read. (14) 

• Such a document needs almost constant up-dating. (45) 

• Publications of this type, become dated too quickly. (92) 

• Material one and a half to two years old at time of publication. (97) 
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For Bibliographies : 



• Now out of date. (12) 

• Useful reference and should be kept up-to-date for teachers. (39) 

• Listings of documents very helpful ... liked the use of fairly 
current research in this report. (61) 

• Please get documents nut closer to publication dates. (62) 

• Build into up-dating service of some sort. (89) 

• Must be updated from time to time. (94) 

• Is quickly made obsolete by newer research. (124) 

3. ORGANIZATION AND CLARITY OF WRITING 

In Table VII-9, evaluation data are displayed on the organization of ideas 
for Reviews and Practical Guidance Papers and of references for Bibliographies. 
Both survey groups rated the organization as better than satisfactory. The 
difference in Reader data between Practical Guidance Papers (2.23) and 
Bibliographies (2.33) is significant at the .05 level. No important differ- 
ences occur between subject areas. 

Table VII-10 shows the judgments on clarity of writing, which are quite positive. 

By recomputing the Reference Mean (as explained on page VII-6), the 2.63 
reported for clarity of writing of documents in the Instructional Content area 
is significantly higher than the new reference mean of 2.48, and the 2.39 for 
documents in the area of Higher Education is significantly lower. In the nine 
cases of Elementary Teachers, their rating of 2.11 on clarity of writing for 
Reviews is low; Principals, with 2.75, had a significantly higher mean. For 
Practical Guidance Papers, Principals were on the low-end, with a significantly 
different 2.18; and Secondary Teachers, with 2.80, were significantly higher 
than the mean. 
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TABLE VI 1-9, 



READERS’ AND SPECIALISTS’ EVALUATIONS ON "ORGANIZATION” 
DIMENSION FOR ALL PRODUCT TYPES 
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TABLE VII-10. READERS’ AND SPECIALISTS’ EVALUATIONS ON "WRITING" DIMENSION 

FOR REVIEWS AND PRACTICAL GUIDANCE PAPERS 



A. READERS’ EVALUATIONS 



Writing (question 10c of Q2-A) . The writing was: 



R (2.51) 
•P (2.53) 



I I I I I 1 I I I I I 1 ! I I I I I 1 I 



1.0 

Hard to 
follow 
at times 



1.5 



2.0 

Moderately 

clear 



2.5 



3.0 

Very 

Clear 



B. SPECIALISTS’ EVALUATIONS 

Clarity of Writing (question 5j of Q4) , 
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Both product types fared well with the Specialists « Ratings in the good/ 
excellent range were assigned to 85 percent of the Reviews and to 79 percent 
of the Practical Guidance Papers. 



Selected comments regarding organization and writing components are given 
below. 



For Reviews: 



• Some redundancies in content. (35) 

• Could be improved by arranging material according to some sort 
of sequence. (36) 

. • Would have been helpful to organize bibliography within 
categories presented by author. (41) 

• Writing style casual and non-academic — makes for easy reading 
by general practitioners not technically trained. (46) 

• Omits publisher, date, number of pages in bibliographic 
references. (47) 

• Basic idea excellent and necessary but writing prosaic and dull. (49) 

t Handling references alphabetically by title is awkward and 
slights the authors. (65) 

• Well done, free from jargon. (65) 



For Practial Guidance Papers: 



• Topical organization of references would be better. (1A) 

t Topical rather than alphabetic listing of references much more 
useful to teachers. (1C) 

• Too much jargon, sentimentality. (2) 

• At times writer appears to be "talking down" to some readers just 
right for many who need it. (40) 




# 
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• Contained several documents within one document .. .good . (40) 

• Presentation of ideas graphically, as well as verbally, useful to 
enhance understanding ... refreshing to find bibliography divided 

and interspersed through paper... most logical to place bibliographic 
reference adjacent to related material. Writing style makes content 
more easily understandable and even enjoyable. (112) 

For Bibliographies ; 

• No excuse for very traditional topics under which language 
arts and reading texts are listed. (8) 

• Clear writing, if somewhat mundane. (39) 

• No system of organization of materials. (42) 

• Would prefer them in alphabetical order by author. (62) 

• Poorly organized .. .mixes units with :eacher guides, ends 
with disorganized miscellany. (Ill) 

• A good job; not cluttered by jargon. (123) 

• Tone of much of this material too varied. . .sometimes objective/ 
scholarly, sometimes almost folksy. (125) 

Some of these comments concern format problems and the pre— publication editorial 
and review process. Responsibility for deficiencies in these areas is, of course, 
not that of the author alone. 

4 . FORMAT 

This element is peripherally related to the areas of organization and writing 
and probably involves the most concrete and distinctive aspects of quality. 
Evaluators were given guidance on interpretation, e.g., typography, 
physical layout, and illustrations, and they showed, through their com- 
ments, considerable interest in these aspects. Since the middle point 

on the evaluation scale— "did not contribute to readability and understanding 
(or use)" — represented a neutral or indifferent attitude coward the effect 
of the format, the data in Table VII-11 indicate that formats can and do 
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TABLE VII- 11. READERS’ AND SPECIALISTS' EVALUATIONS ON "FORMAT" DIMENSION FOR 

ALL PRODUCT TYPES 




contribute positively to the perceived quality of the products. Readers were 
dependent upon their memories for answering this fairly speciiic question 
(although there was evidence from individual comments that many respondents 
had copies in hand when they answered the questionnaires). On the other hand, 
all Specialists had copies in hand and they distributed their ratings from fair 

to excellent. 

The comments in this area are highly illuminating, particularly when viewed in 
relation to the specific products under study. 



For Reviews : 



• Some lines uneven; typing errors; type a bit small. (15) 

• Sub-topic headings would have provided greater clarity. (34) 

• Articles should be concluded without continuing to later pages. (35) 

• Side heads would facilitate ease of reading. (37) 

• Format section headings in bold or different type. (41) 

• Could use more presentations via charts and graphs. (46) 

• Stodgy and conventional looking.. .difficult to read because 
of squeezed typography. (66) 

• Appears crowded. .. lacks illustrations. (108) 



For Practical Guidance Papers: 



• IRA publication earier to handle and work with. (1) 

• Should make effort at uniform editions of publications. (2) 

• Table of contents would facilitate usage. (4) 

• Sectioned introductions present key questions or outline 
major ideas to be covered — helps in reading a very unattractive 
manuscript. (6) 

• Outline format causes it to read a bit choppy. (26) 
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• Lack of paragraph headings and captions are a limitation 
and prevent maximum use of excellent materials. (40) 

• Copy difficult to read... I believe that we need such a 
document but "Buyer Beware." (45) 

For Bibliographies : 

• Author index or combined analytical index would add to usefulness. (11) 

• Inadequate margins top and bottom; looks cheap and makes reading 
difficult; change of type is poor. (12) 

• Variation in margins and many pages not numbered . (18) 

• Illustrations detract from document. (39) 

• Needs cross- indexing by subject areas, as there is overlap 
in clearinghouse products. (86) 

• Formidable format .. .many Title III ESEA proposal writers should, 
but would not, bother with it. (101) 

• Format uninteresting, tedious, somewhat monotonous. (Ill) 

5. CHOICE OF AUTHOR AND ACCURACY IN REPORTING 

These two more sensitive issues were addressed only by the Specialists. Table 
VII-12 shows that, in each case, 50 percent of the ratings were in the 
good/excellent categories. 

Evaluation of the choice of author presented Specialists with a difficult task. 
Those not familiar with the author based their ratings on the general quality of 
the job. Also, the questionnaire item was not structured to accommodate a more 
positive evaluation of one coauthor over another. Nevertheless, some evaluators 
indicated definite differences in their evaluations of two or more authors who 
were clearly identified as chapter or section writers. The question was not 
always seen as applicable in the case of bibliographies, particularly those where 
the compilers were not identified or were members of the clearinghouse staff. 
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TABLE VI 1-12. 



SPECIALISTS’ EVALUATIONS ON "AUTHOR SELECTION” AND 'ACCURACY 
DIMENSIONS OF ALL PRODUCT TYPES 








Ju. 



X 
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Again, some 



of the comments aid in interpreting these ratings. 



/> 



For Reviews : 

• Author a recognized authority in this field of endeavor. (37) 

• Although authors are both non-educators, have discussed a 
difficult subject in manner appropriate and useful. ... (38) 

• Neither author has contributed significant publications on the 
topic... this may not be crucial, but national visibility might 

be helpful. (47) 

• So much more should be taid. • .leaves incorrect impression. (50) 

• Do not agree with many distinctions drawn in this piece. • • * (55D) 

• Author informed, yet not a special pleader for FLES. (84) 

For Practical Guidance Papers : 

• Page dealing with language disadvantaged children filled 
with inaccuracies. (1A) 

• Biased report; lacks relationship to practice. (25) 

• Accurate, but not specific or complete enough. (26) 

• Author probably the best person to write this report. (81) 



For Bibliographies : 



I have never read anything by these authors in the professional 
literature. (8) 

Author very knowledgeable. (12) 
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No author indicated. (Ill) 



• A prestigious individual would have lent needed credibility 
to this highly sensitive topic. (114) 

• Author a professional bibliographer. . .perhaps authority on 
topic would have been better choice. (115) 

In general, few comments were made concerning inaccuracies in reporting; rather, 
they were targeted toward omissions or evidences of biases. The author- related 
comments are fairly typical of the range of opinions obtained. 
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c. 



EVALUATIONS OF DIMENSIONS OF UTILITY 



The concept of utility Is defined In this study by several types of items: 
relevance, need, degree of usefulness, and comparative usefulness. We used the 
definition, fully recognizing that utility might well be a simple outcome of 
quality, i.e., a quality document is a useful one. However, there are some 
indications, particularly with bibliographies, that use may be a fairly 
independent variable and that the need for a document on a particular topic may 
supersede the need for a quality document. This hypothesis has not been tested 
by any special analyses in this study, and pursuit of its validity must be left 
to some future study. 

It is in this area of utility that the survey instruments diverge more widely 
in structure and content. Some of these differences are highlighted below: 

• The i'tem on use in the Users’ Evaluation Questionnaire for 
substantive papers (Q2-A) inquires about degree of useful- 
ness for various specified purposes. In the bibliography 
version (Q2-B) , the item simply asks how the bibliography 
was used. 

• In the Specialists’ Questionnaire, the major item on use is 
posed after the Specialists have been asked whether they 
would recommend documents to colleagues • Since the Specialists 
were asked to imagine the variety of ways in which products 
might be used — not how they were used — the number of uses might 
be expected to be somewhat broader from this group. 

There are, however, three directly comparable questions in the three evaluation 
survey instruments, on relevance, need, and comparative usefulness. In addition, 
we report data on potential relevance and need, a commonly phrased item for 

Non-users (Q3) and Non- readers (Q2) . 
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RELEVANCE AND NEED 



Special ists . For Specialists, the item on relevance was included as a test of 
validity for document assignments. On the assumption that a certain bias of 
indifference might enter into an evaluation of a jion-relevant document, Specialists 
were instructed to return documents unevaluated if they were "not at all relevant". 
The "need" question was phrased in such a way as to obtain a general indication 
of the "sense of urgency" in the field for topics covered and by product types. 

Data from Specialists for both questions are presented in Table VII-13. The 
high percentages on the relevant end of the scale support the validity of the 
document assignment process. Although presented in percentages, the results 
for the "need” question, when converted to means, indicate that Specialists 
believed the need to be generally greater than that recognized by the Readers, 
shown in Table VII-14. Means for Specialists are: 2.56 for Reviews; 2.43 for 
Practical Guidance Papers; and 2.48 for Bibliographies. 

Reader, Non-Reader, and N on- User Data on Relevance . Ratings on the "relevance" 
question (see Tables VII-14 and 15) by General Field Survey participants are 
probably most useful as a measure of success in this study's document assignment 
procedures. Documents were assigned individually on the basis of general 
professional interest information (and other background data reported in 
Screening Questionnaire returns) and the respondents generally, and simply, 
indicated that they were reading or were potentially interested in products 
covering these areas. Further interpretation raises the issue of timeliness 
because, as pointed out by some respondents, what was relevant at the time of 
publication may not be relevant now, or vice versa. Also, respondents indicated 
that shifts in reponsibility and interests or new trends in their fields would 
affect the relevancy of documents over a period of time. 
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TABLE VII-13 . SPECIALISTS 1 EVALUATIONS ON "RELEVANCE" AND "NEEDS" DIMENSIONS 



FOR ALL PRODUCT TYPES 




TABLE VII - 14 . READERS' EVALUATIONS ON 
ALL PRODUCT TYPES 



"RELEVANCE" AND "NEED" 



DIMENSIONS FO. 



READERS' EVALUATIONS 



Relevance (question 6 of Q2-A and -B) . How relevant was the topic to 
your general professional interests? 



I I i I I 1 I I I I t I I 1 



1.0 



1.5 



2.0 



2.5 



P (2.67) 
R (2.72) 



fr-B C2.77) 

I I 11 1 JJ 



3.0 



Not at 
all relevant 



Somewhat Relevant 

relevant 



Need (question 7 of Q2-A and -B) . As of the publication date, how 
great was your need for a good document on this topic? 



I 1 l l l 1 l 

1.0 1»5 

Not at all great: 

I had no special 
need for it. 




2.0 

Moderately great: 
The topic is of 
continuing impor- 
tance to me. 



(2.33) 

(2.35) 

(2.39) 

I I I I I 

3.0 

Very great : 

I had an immediate 
reed for a document 
on tnis topic . 
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TABLE VII- 15. JUDGEMENTS ON POTENTIAL "RELEVANCE” OF DOCUMENTS REPORTED BY 

NON-READERS (Q2) AND NON-USERS (Q3) 




Comments were not generally directed toward the issue of relevance, except by 
those respondents who had heard about a document but had not read or skimmed 

it. They indicated, for example, that they knew it was available if and when 
it became particularly relevant — or needed. 



Rea der, Non-Reader, and Non-User Data on Need . For purposes of this study "need” 
defined as the sense of urgency for or timeliness of the products. The response - 
choice points to the question of "As of the publication date, how great was your 
need for a good document on this topic?" were: 

• Not at all great; I had no special need for it. 

• Moderately great; the topic is of continuing importance to me. 

• Very great; I had an immediate need for a document on this topic. 

Thus in a sense, the three-point scale contains two positive ratings and one 
negative, unless a policy statement commits the analysis effort toward more 
urgent topics, rather than topics of continuing importance in the field. 

As shown in Table VII-14, there are no significant differences for Readers 
among product types, but the clustering of ratings toward middle -choice points 
suggests that products are judged as addressing topics of continuing importance 
in the field. Extensions of this interpretation could be made on the basis 
of assumptions about the respondent populations' less-urgent need for infor- 
mation or about the reliance of analysis products on somewhat well-established 
trends in research and practice. 

The need by Non-readers and Non-users for products brought to their attention 
on this study is reported in Table VII-16. The similarity between the two 
populations is particularly striking. It suggests that the Non-user is, indeed, 
a "potential user," with information needs similar to those of the user popu- 
lation in this study. This interpretation is borne out in the following section 
on why products are not read by those who have heard about them. 




1 ' ‘ 



J 
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TABLE VII-16. JUDGEMENTS ON POTENTIAL "USEFULNESS" OF DOCUMENTS REPORTED BY 

NON-READERS (Q 2) AND NON-USERS (Q3) 



A. NON-READERS [ (FROM Q2) EVALUATIONS 

Need (question 3 of Q2 A and B) . As of the publication date, how great 
was your need for a good document of this type on this subject? 



100 % 



Would probably 
have been of little 
or no use 



50% 



0 % 



( 22 %) 



( 21 %) 



(23%) 



R 



B 



Would probably have 
been of some use 



(39%) 



(37%) 



(35%) 



Would probably 
have been very 
useful 



(38%) 



(34%) 



R 



(29%) 



r 



B 



B. NON-USERS 1 (FROM Q3) EVALUATIONS 

Need (question 3b of Q3) . As of the publication date, how great was your 
for a good document of this type on this topic? 



100 %. 



50% 



or 



Would probably 
have been of little 
or no use 



(31%) 



R 



(23%) 

— 1 



(29%) 



B 



Would probably have 
been of some use 



(34%) (34%) 



(42%) 



R 



B 



Would probably 
have been very 
useful 



(34%) 



(42%) 



<r 



(28%) 



B 



3 

ERIC 




t 
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Reasons for not Reading * Beyond the issue of need and potential relevance is 
the question of why respondents, who reported having previously seen or heard 
about a document, had not read it. It is the population of Non-users that bears 
particularly close examination for drawing inferences about their being '’potential" 
product users. Data on this question, shown in Table VII-17 are both dramatic 
and instructive. Non-users, who are presumably not well acquainted with the 
educational dissemination system, appear to believe that products are not 
readily accessible. Non-readers, however, report a variety of reasons for not 
reading documents. This latter group indicates some iiculty in obtaining 
copies but al^o shows some evidence of being more discriminating in what they 
do read. Also, their knowing about the availability of products may be suffi- 
cient, for they can read it when needed. Some of the Non-reader comments, cited 
oeJ.ow, obtained from this question help to illustrate further some "non-use" patterns. 

For Reviews: 



• Had other priorities. (13) 

• Postponed getting to it till strong interest had passed. (15) 

• Loaned it to colleagues with specific interest in area. (16) 

• I may have skimmed it... very difficult to recall all these 
documents. (23) 

\' - 

• Have used other ERIC documents on the same subject 1 (36) 

• Read some sections. (37) 

• Limited literature budget. (38) 

• Have not made an effort to keep up in this research field... my 
professional area is now different. (41) 

• Discussed paper with person who heard it presented. (41) 



• Did not take time to seek out... a current awareness system needed... 
such as sending copies of fly sheets. (68) 
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TABLE VII-17 . 



REASONS FOR NOT READING PRODUCTS PREVIOUSLY SEEN OR HEARD OF 
REPORTED BY NON-READERS (Q2) AND NON-USERS (Q3) 









For Practical Guidance Papers: 



• Have only recently developed a need for it. (3) 

• Forwarded to Department Head. (6) 

• Have no chance to change my laboratory. (81) 

• Had other material on subject readily available. (81) 

• We are reorganizing our Economics curriculum, but teachers did 
not want to use it, as Economics had been taught in our high 
schools as required courses for many years. (112) 

For Bibliographies : 

• Have seen no reference to this one, although I know earlier 
documents on disadvantaged. (28) 

• Did not deal specifically with questions I had to answer. (31) 

• Need better distribution and availability to the classroom 
teacher. (86) 

• Had completed the bibliography when I came across this... 
used it to check my own list. (95) 

• As I recall, procedure for getting copies was so complicated that 
I assumed the required time would not be worth the result. (103) 

2. PURPOSES OF USE AND DEGREE OF USEFULNESS 

Different purposes for which substantive papers were used by Readers, and 
their degree of usefulness, are reported in Table VII-18. 

A problem in interpreting these data occurs as a result of the scoring on this 
particular question. The neutral response, "did not use for this purpose", 
was inadvertently included in the scale, and therefore, the results do not 
indicate clearly the relation between use and degree of usefulness. However, 
the order of uses does suggest that it is perhaps necessary to target documents 
specific uses by intended audiences. 




C. . 
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TABLE VII-18. READERS’ EVALUATIONS ON "DEGREE OF USEFULNESS" DIMENSION FOR 

REVIEWS AND PRACTCCAL GUIDANCE PAPERS 



Degree of Usefulness (question 11 of Q2-A) . Please indicate how useful the 
document was to you for each of the purposes listed. (If you did not use 
the document for a stated purpose, check the last column ["did not use for 
this purpose"] . 



PURPOSES OF USE 


DEGREE OF USEFULNESS 






To obtain an overview 




of the topic 






To update my knowledge 




about a subject al- 


m 1 1 • t/ J 


ready known to me 




To identify relevant 


R(2 . 36) 


literature reier 
^nces 




To look up needed 


- R(2 . 20) 


facts 


4 . P(2 .24) 


|To obtain knowledge 


R(2.14) 


about a new sub~ 




ject 




To identify indi- 


R(2.13) 


viduals working 


P(2.12) 


in the area 




- 


1 1 1 1 1 1 1 1 1 1 1 II 1 1 1 1 1 L 



1.0 1.5 2.0 2.5 3.0 

Not at all Somewhat Very 

useful useful useful 
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Some differences occur by general subject area. For example, there are 

differences in the usefulness of Practical Guidance Papers for updating know- 

' « • 

ledge in the areas of Special and Other Educational Groups (with a mean of 

‘ „ * 

2.50) and Higher Education (with a mean 0^2.27). 

f * ** 



1 » 

Table VIT-18 shows the response percentage in descending order of frequency, 
not as the choices appeared in the questionnaire. It is interesting to compare 
this order (from the first, "tO|Obtain an overview'*, to the last, **to identify 
individuals") with patterns within user groups. In the display below, 
variations in the order of use for the overall population are indicated for 
particular user groups. 



For Reviews 

Obtain overview 
Update knowledge 
Identify literature 
Look up facts 
Obtain new knowledge 
Identify individuals 



Researchers 

^ 1 

3 
2 

> 5 

6 

4 



Elem. Teachers Supt . 

1 1 

2 2 

3 3 

3 4 

2 5 

4 5 



For Practical Guidance Papers 



Researchers Elem. Teachers Supt . 



Obtain overview 
Update knowledge 
Identify literature 
Look up facts 
Obtain new knowledge 
Identify individuals 



1 

2 

4 
3 

5 

6 



1 

1 

2 

1 

3 

3 



1 

2 

3 
5 

4 

5 
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Researchers differed slightly and in expected ways, in their uses of Reviews; 
they used these products first to "identify literature" and second to "obtain 
overview". The most interesting differences occur with Elementary Teachers, 
who appear to use both Reviews and Practical Guidance Papers in differenct ways, 
i.e., Reviews, "to obtain new knowledge", and Practical Guidance Papers, "to 

look up facts". 



The potential usefulness of products .is reported for Specialists in Table VII-19 
Specialists were first asked to indicate if they would recommend products to 
their colleagues, and the usefulness data represent only those cases in which 
the response was "yes". The high yes-response rates are shown below: 



Reviews . 

Practical Guidance Papers 

V * V 

Bibliographies 



191 (92%) 
78 (83%) 
114 (88%) 



Specialists agreed with Readers thfct the principal usefulness of the substantive 
papers was in oabtaining an overview of a topic. The order (looking just at per- 
centages in the "very useful" column) varies from that point on, partly due to 
the addition of a new category, "obtain practical guidance," but also because 
of an apparently differenct perspective on the the usefulness of the products 
for obtaining new knowledge. This perspective from the Specialists, perhaps, 
helps to clarify the results from Readers. The products may very well be useful 
for obtaining new knowledge, and Readers in general, may simply not be using 

them in that way. 



As noted earlier, Readers were asked to indicate their use ot Bibliographies 
from a simple checklist, irfeir responses are represented in Table VII-20. 
Although the sequence is not particularly surprising, it demonstrates 
priority needs and uses of this product type. 



some 



TABLi: VI 1-19. POTENTIAL USES OF THOSE PRODUCTS 

SPECIALISTS 



RECOMMENDED TO COLLEAGUES 



BY 





Not at 
all useful 


Somewhat 

useful 


u 


Very 

seful 




Purpose of Use 




R 


P 


B 


R 


P ! 


B 


R 


P 


B 


Obtain overview 


1% 


4% 


7% 


20% 


24% 


27% 


80% 


72% 


61% 


Look up facts 


12% 


17% 


20% 


36% 


47% 


47% 


31% 


36% 


29% 


Identify relevant 
literature 


3% 


17% 


0% 


32% 


32% 


28% 


64% 


46% 


71% 


Identify individuals 
or institutions 


8% 


21% 


5% 


41% 


32% 


49% 


51% 


42% 


46% 


Update knowledge 


8% 


9% 


9% 


47% 


45% 


45% 


45% 


44% 


44% 


Obtain new knowledge 


11% 


12% 


17% 


36% 


39% 


48% 


51% 


50% 


30% 


Obtain practical 
guidance 


9% 


5% 


19% 


45% 


26% 


40% 


44% 


69% 


| 38% 


Other 


0% 


0% 


0% 


1% 


4% 


1% 


6% 


8% 


_Ii 
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TABLE VII-20. READERS 1 REPORT ( USE 05 BIBLIOGRAPHIES 
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The substance of these uses is more clearly understood from comments by Readers. 
These are included xn Section D on "impact”. 

3 . COMPARATIVE USEFULNESS 

Percentages from the Specialists were converted to means in order to highlight 
comparisons with Readers on this summary question. Results from both survey 
groups are presented in Table VII-21. Although Specialists were not as 
positive, the total picture is very favorable, particularly given the assumption 
that Specialists were indeed familiar with "documents of the same type." 

The most striking difference lies in the significantly higher evaluation of 
bibliographies by the Reader Group. This 2.70 can be interpreted to mean that 
among the available bibliographies, NCEC-produced ones are particularly useful. 

It might also suggest that NCEC units are principal suppliers of this particular 
product type, whereas, they have more serious competition from other sources 
for the other two product types. 

For Reviews, the high and low groups are the Special Educators (2.83) and 
Counselors (2.40). Elementary Teachers were also low, with 2.44. For Practical 
Guidance Papers, the high and low groups are College Admins trators (2.79) and 
Counselors (2.24), respectively. For Bibliographies, the Special Educators 
were also high with a mean of 2.94, and Vocational Educators and Other 
Administrators, with means of 2.40 each. 

Few comments pertained specifically to this item. In some instances, respondents 
indicated "I know of no document that is comparable to (a PREP Report) (Teaching 
Exceptional Children Journal) ." 
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TABLE VII-21 . READERS’ AND SPECIALISTS’ EVALUATIONS ON ’’COMPARATIVE 

USEFULNESS" OF PRODUCTS 



A. READERS ’ EVALUATIONS 



Comparative usefulness (question 13 of Q2-A and Q2-B) . In general, how would 
you compare this document with other documents of the same type? 



i P (2.52) 

i R (2.58) 

|— B (2.70) 

1 I I I l 1 M l I I l I l l I l l l l I 

1.0 1*5 2.0 2.5 3.0 

Usefulness too Not unusually Very useful 

limited to justify useful, but worth 

its publication having available 



B. SPECIALISTS’ EVALUATIONS 



Comparative usefulness (question 7 of Q4) . In general, how would you compare 
this document with other documents of the same type? 



P (2.36) 
B (2.44) 
R (2.57) 



1.0 1.5 2.0 2.5 3.0 

Usefulness too Not unusually Very useful 

limited to justify useful, but worth 

its publication having available 
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D. 



EVALUATIONS ON THE DIMENSION OF IMPACT 



The impact question was addressed as an extension of the utility dimensi >n. It 
was handled in two ways to accommodate differences between the two major types 
of products. For substantive papers (Reviews and Practical Guidance Papers), 
the questionnaire item was: "As a result of reading the document, did you use 

the information or the document in any of the following ways?" For Bibliographie 
the issue was treated in . format-related way. The item was: "As a result of 

using this document, did you examine any of the documents cited? If yes, was the 
content of the document(s) what you had been led to expect by the content of 
the bibliographic ref erence(s) ?" 

Results cor all product types are displayed in Table VII-22. (Specialists were 
not asked to respond to an impact-related question because their assigned role 
was once-removed from that of the general user.) The most frequently reported 
impact was one of general application to work. Practical Guidance Papers were 
used slightly more as a source for imparting advice to ethers, and for making 
decisions. The most supportive finding for the quality judgments just discussed 

is the high percentage of cases in which documents were passed on to colleagues 
to read. 

The strongest indications of impact came from comments by Users. Examples of 
these are presented below. 

For Reviews : 



• Helped in providing tecnnical assistance to several community 
colleges and university extension divisions. (15) 

• Used data to develop and plan residential institute. (16) 

• Used in adult educational class and seminar. (19) 

• Used in advising researchers in developing research design for 
improving professional training programs for adult educators. (19) 



TABLE VII- 22. READERS' REPORT ON ’'IMPACT” DIMENSION FOR ALL PRODUCT TYPES 



Impact t or Reviews and Practical Guidance Papers (question 12 of Q2-A) . As a 
result of reading the document, did you use the Information or the document in 

any of ti;°. following ways? 



IMPACT 



Applied the information 
in my own work 

Used the information to 
give advice to other 
people 

Passed the document on 
to a colleague (s) to 
read 

Examined other docu- 
ments 

Used the facts or re- 
commendations to 
make a decision 

Consulted with author (s) 
or other persons iden- 
tified in the document 



PERCENTAGES OF CASES 



—>(69%) R 
-(65%) P 



(46%) R 
— (50%) P 



•(42%) R 
— (49%) P 



(32%) ^ 

-(27%) P 



(19%) R 
_(23%) P 



-(8%) R 

.(6%) P 



0 % 



50% 



100 % 



Impact for Bibliographies (question 12 of Q2-B) . As a result of using this document, 
did you examine any of the documents cited? 

YES =76% 

If yes , was the content of the document (s) what you had been led to 
expect by the content of the bibliographic reference? 



YES = 73% 



NO = 27% 
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• Found helpful for my own information. (23) 

• Used it as a basis for helping teachers to analyze their behavior. (35) 

• Helped me think through ideas about writing own paper on topic. (37) 

• Useful for self-evaluation. (41) 

• Helped provide background for work on Governor's commission. (46) 

• Our department is redesigning the curriculum for school administrators, 
and this document helped form my judgments regarding curriculum 
matters. (53) 

• Helped in aspects of dissertation. (59) 

• Gave background for design of a vocational education program. (139) 

For Practical Guidance Papers: 



• Used in my work in new Title III reading project. (1) 

• Helped me prepare for working with teachers on the state of the 
art in teaching disadvantaged. (3) 

• Recommended to V.P. for consideration in staffing. (4) 

• Was helpful in training project staff members in Title III project. (40) 

• Very useful for training teachers. (63) 

• Used to place children in a variety of reading materials .. .quick 
reference for me. (92) 

• Was able to select and order samples of curriculum guides ^r 
reference for myself and my school. (133) 

For Bibliographies : 



• Provided stimulus and interest as well as knowledge enabling teacher 
to utilize creative writing with remedial reading youngsters. (8) 

• Extremely helpful in graduate studies in the education of adults. (12) 
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• Invaluable in developing R & D projects, in designing educational 
conferences, and in advising doctoral students* (18) 

• Was helpful in research for developing a new project. (21) 

• Used in connection with our own Title VIII project. (33) 

• Great while studying junior year... gives background for job 
selection. (56) 

• Sent copy to museum director in Canada — contained reference that 
proved valuable. (57) 

• My students have used it regularly and found references in line 
with expectations. (79) 



V 
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E. 



EVALUATIONS BY PRODUCT TYPE AND LEVEL OF EFFORT 



Although it was not in the scope of this project to study the "development 
environment" of NCEC products, it was felt that a fair assessment must relate 
in some way to measures of investment in time and cost. It was also important 
to use such measures in exploring the issue of cost-effectiveness so that 
factually based guidelines for allocating resources can eventually be developed. 
The general question asked was: Does increased level of effort produce a 

better quality product? 

For Reader data, this question was expanded beyond quality to include in 
addition, effects of level of effort on utility and impact. Reference tables 
containing results of this complete analysis are presented in Tables VII- 
41, 46, and 51 at the conclusion of this chapter. These tables show means 
and percentages for each quality, utility, and impact item by level-of-ef fort 
groupings within product types. The data from these tables were trans- 
lated into a simple, visual display, in Table VII-23, that is based on a 
comparison of the means and percentages between low- and high-level-of-ef fort 
products . 

The three columns in this table summarize the number of instances in which 
ratings for high-level-of-ef fort products in Tables VII-41, 46, 51 x^ere the 
same as, lower than, or higher than those for low-level-of -effort products. 

This simple summary serves roughly as a balance sheet. The differences represented 
in this "balance sheet" are insignificant on the whole, but do reflect some 
pattern of difference in the impact-related areas, particularly for Reviews 
and Practical Guidance Papers. 

One final analysis on the effort/quality issue was performed. Reader data 
were analyzed by user groups for each level-of-ef fort groups of documents by 
product type. Results of this three-way analysis are displayed in Tables VII- 
24 through 29 for six cuality/utility items: coverage, up-to-dateness, format, 

relevance, need, and comparative usefulness. 



TABLE VII-23. APPARENT EFFECTS OF HIGHER LEVEL OF EFFORT PRODUCTS (A SUMMARY 

OF COMPARISONS BETWEEN HIGH EFFORT AND LOW EFFORT RATINGS IN 
TABLES 43, 48, 53) 





Ratings on QUALITY of High Effc 


>rt Products 


Lower than 
Low Effort 


Same as 
Low Effort 


Higher than 
Low Effort 


Reviews 


5 


1 


1 


Practical Guidance 
Papers 


4 


0 


3 


Bibliographies 


1 


0 


5 


TOTAL 


10 


1 


9 





Ratings on UTILITY of High Effort Products 


Lower than 
Low Effort 


Same as 
Low Effort 


Higher than 
Low Effort 


Reviews 


2 


1 


6 


Practical Guidance 
Papers 


2 


0 


7 


Bibliographies 


7 


0 


2 


TOTAL 


11 


1 


15 





Ratings on IMPACT of High Effort Products 


Lower than 
Low Effort 


Same as 
Low Effort 


Higher than 
Low Effort 


Reviews 


0 


1 


5 


Practical Guidance 
Papers 


0 


0 


6 


Bibliographies 


0 


0 


2 


TOTAL | 


0 


1 


13 
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Top percentages in each cell in these six tables are those for the most positive 
response choices (e.g., "relevant"). Although these results reveal some 
Interesting patterns among user groups and across product types, a major inter- 
pretation problem lies in the small sample size in almost all cases. (Those 
cells with no entries indicate that no evaluation data were available for that 
particular combination of variables.) 

These data were studied in cases where at least 10 ratings were available for 
both high- and low-effort products. Of the 23 cases, none is significant at 
the .05 level or better. In a separate step, we examined cases where there 
were proportionate differences in ratings of at least 2QX and still found no 
clear relation between high effort and high quality. 

These inconclusive findings suggest several things. One is that the level-of- 
effort measure may need to be expanded upon, which in turn suggests the need 
for a uniform cost-accounting system throughout NCEC. Second, there may be 
other elements in the creation cycle (for example, the decision-making process 
that precedes creation or the prepublication edit/review process) that 
contribute more significantly to quality ratings than does the level of effort 
(i.e., professional man-hours). 
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TABLE VII-24. PERCENTAGE OF FAVORABLE AND UNFAVORABLE RAT; 

BY USER GROUP AND LEVEL OF EFFORT* 





Reading 

Specialist 


' Special 
Educator 


Vocational 

Educator 


Supervisor 


High 

Effort 


Medium 

Effort 


Low s, 
Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Eftu'-t 


Low 

| F.ffot 




OZ 




OZ 




OZ 


OZ 


20Z 


OZ 


0% 


OZ 


OZ 1 


O? 


REVIEWS 


25% 




50 Z 


¥ 


67Z 


44Z 


60% 


42Z 


60 Z 


25Z 


69Z 


88: 


PRACTICAL 


OX 


OZ 






OZ 


100Z 


0% 


OZ 




OZ 


OZ 


0, 


GUIDANCE 


40Z 


60Z 






50Z 


OZ 


502 


502 




50 Z 


33Z 


897 


PAPERS 




























OZ 




OZ 


OZ 




OZ 


OZ 


OZ 




OZ 


33% 


o: 


BIBLIOGRAPHIES 


33Z 




29Z 


, 50Z 




56Z 


OZ 


50Z 




67Z 


33Z 


so: 





Researcher 


Instructional 
Resources Specialist 


Program 

Specialist 


Principal 






High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 




Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effo 


REVIEWS 


OZ 


1Z 


1Z 


0% 


0% 


0% 


11% 


4% 


3% 


20% 


0% 


0% 


40 Z 


48Z 


43Z 


33% 


60% 


56% 


63% 


48% 


56% 


80% 


33% 


78% 


PRACTICAL 


7% 


11% 


0% 


f o% 


82 


0% 


3% 


0% 


5% 


0% 


0% 


0% 


GUIDANCE 

PAPERS 


27% 


32Z 


25% 


? 76% 


62% 


100% 


61% 


42% 


57% 


33% 


0% 


0% 




OZ 


OZ 


0% 


5% 


7% 


3% 


0% 


42 


3% 


0% 


0% 


0% 


BIBLIOGRAPHIES 


40Z 


71Z 


35% 


42% 


67% 


40% 


62% 


40% 


46% 


80% 


0% 


67% 



i 





Secondary 

Teacher 


\ 


College 

Professor 


College 

Administrator 


Superintendent 




High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effor 


REVIEWS 


0% 


0% 


OZ 


OZ 


3Z 


2Z 


OZ 


3Z 


32 


02 


OZ 


0Z 


CUif nf u 


100% 


93% 


60Z 


59Z 


55Z 


54Z 


47Z 


4?Z 


412 


69Z 


67Z 


55; 


PRACTICAL 


0% 


0% 


OZ 


20Z 


3Z 


4Z 


OZ 


OZ 




OZ 


50Z 


0Z 


GUIDANCE 

EA2ERS 


502 


0% 


27Z 


47Z 


57Z 


42Z 


43Z 


50Z 




54Z 


OZ 


0, 




02 


OZ 


OZ 


OZ 


4Z 


2Z 


OZ 


50Z 


02 


OZ 


OZ 


OZ 


BIBLIOGRAPHIES 


542 


100Z 


42Z 


63Z 


54Z 


61Z 


1002 


02 


332 


OZ 


75Z 


56Z 



i f 

In each cel the top percent indicates unfavorable ratin 
percent favorable ratings; intermediate ratings are not r 
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1-24. PERCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF "COVERAGE" 



BY USER GROUP AND LEVEL OF EFFORT* 



Special 

Educator 


Vocational 

Educator 


Supervisor 


Counselor 






Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


t 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 




0% 


02 


202 


02 


02 


02 


02 


02 


02 


02 


02 




672 


442 


602 


422 


602 


252 


692 


882 


252 


422 


502 




02 


1002 


02 


02 




02 


02 


02 


02 


72 


172 




502 


02 


502 


502 




502 


332 


892 


602 


292 


172 






02 


02 


02 




02 


332 


02 


02 


202 


02 






562 


02 


502 




672 


332 


502 


502 


02 


1002 



instructional 
jurces Specialist 


Program 

Specialist 


Principal 




Elementary 

Teacher 




Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Lov 


rt 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort _ 


hi 


02 


02 


112 


42 


32 


202 


02 


02 


02 




142 


** 


602 


562 


632 


482 


562 


802 


332 


782 


1002 




292 


ft 

f* 


82 


02 


32 


02 


52 


02 


02 


02 


02 


02 




X 


622 


1002 


612 


422 


572 


332 


02 


02 


1002 


1002 




r 


72 


32 


02 


42 


32 


02 


02 


02 


02 


02 


02 




672 


402 


622 


402 


462 


80% 


02 


672 


1002 


1002 


752 



College 

Professor 


College 

Administrator 


Superintendent 


Other Administrative 
Positions 




Medium 


Lav 


High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


. Low 


rt 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 




32 


22 


02 


32 


32 


02 


02 


02 


02 


02 


02 




552 


542 


472 


432 


412 


692 


672 


552 


1002 


332 


252 




32 


42 


02 


02 




02 


502 


02 


02 


02 






572 


422 


432 


502 




542 


02 


02 


482 


02 




- 


4% 


22 


02 


502 


02 


02 


02 


02 


132 


02 


02 




542 


612 


1002 


02 


33% 


02 


752 


562 


132 


502 


332 



cell, the top percent indicates unfavorable ratings and the bottom 
favorable ratings; intermediate ratings are not represented. 
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Secondary 

Teacher 


College 

Professor 


College 

Administrator 




Su] 


per intend 


lent 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


^igh 

Effort 


Me a turn 
Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


REVIEWS 


0 % 

100 % 


14 % 

79 % 


0 % 

100 % 


5 % 

85 % 


3 % 

89 % 


5 % 

87 % 


0 % 

93 % 


t " 

89 % 


3 % 

86 % 


0 % 

92 % 


0 % 

89 % 


0 % 

91 % 


PRACTICAL 

GUIDANCE 

P A ppOC 


0 % 

50 % 


0 % 

0 % 


0 % 

93 % 


20 % 

60 % 


13 % 

77 % 


13 % ' 

88 % 


0 % 

71 % 


0 % 

83 % 


— 


0 % 

96% 


0 % 

50 % 


oz 

100 % 


r AJr 

BIBLIOGRAPHIES 


0 % 

85 % 


0 % ! 
100 % 


17 % 

67 % 


3 % 

93 % 


4 % 

79 % 


2 % 

89 % 


0 % 

100 % 


0 % 

100 % 


11 % 

89 % 


0 % 

100 % 


0 % 

| 100 % 


0 % 

89 % 



*In each cell, the top percent indicates unfavorable rating, 
percent favorable ratings; intermediate ratings are not re* 
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ERIC 



ERCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF "UP-TO- 
’ ATENESS" BY USER GRO UP AND L EVEL OF EFFORT* 





V ch'i' 1 t ; 1 

Educ iior 


Supervisor 


Counselor 


1 Low 
j r.ffort 


High 

Effort 


fled; <i;:i 
Ef i 'rt 


1a*V 

Lf 1 <rt 


High 
Kf Fort 


Met. tu:-. 1 
r.f'o; <. 


Low 

.if fort 


KT 


Medium 

Effort 


Low 

Effort 


i 0% 

< 67% 


707 

80% 


V 

8 ♦ 


01 

100% 


0% 

75% 


‘)T“ 

< 2 . 


0% 

88% 


0% 

75% 


8% 

83% 


0% 

79% 


>5 0% 


0% ’ 






0% 


ox 


ox 


01 


0% 


0% 


100% 


75% 


1001 




75% 


100% 


89% 


60% 


93% 


83% 


6% 


0% 


2?% 




0% 


33% 


0% 


50% 


0% 


0% 


75% 


100% 


75% 




83% 


33% 


83% 


50% 


20% 


50% 



iU.1 

aiist 


Program 

Specialist 


Principal 


Elementary 

Teacher 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


High 


Medium 


Low 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


0% 


5% 


4% 


8% 


0% 


0% 


OX 


0% 




0% 


75% 


84% 


78% 


83% 


80% 


83% 


100% 


100% 




57% 


0% 


8% 




, 26% 


0% 


0% 


0% 


0% 


0% 




100% 


87% 


77% 


68% 


78% 


100% 


100% 


100% 


100% 




3% 


0% 


8% 


8% 


0% 


0% 


0% 


0% 


OX 


0% 


90% 


92% 


76% 


92% 


100% 


100% 


100% 


100% 


100% 


88% 





College 

Administrate 


r 


Superintendent 


Other Administrative 
Positions 


Low 

Effort 


HI CO 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High | 
Effort 


Medium 

Effort 


Low 

Effort 


. 5% 


0% 


6% 


3% 


0% 


0% 


0% 


0% 


0% 


25% 


87% 


93% 


89% 


86% 


92% 


89% 


91% 


100% 


50% 


S0% 


13% 


0% 


0% 




0% 


0% 


0% 


3% 


100% 




88% 


71% 


83% 




96% 


50% 


100% 


83% 


0% 




2% 


0% 


0% 


11% 


0% 


0% 


0% 


13% 


0% 


0% 


89% 


100% 


100% 


89% 


100% 


100% 


89% 


50% 


50% 


67% 



e top percent indicates unfavorable ratings and the bottom 
e ratings; intermediate ratings are not represented.' 
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TABLE VII-26. PERCENTAGE OF FAVORABLE AND UNFAVORABLE RATI? 

BY USER GROUP AND LEVEL OF EFFORT* 





Reading 
Spec i a list 


Special 

Educator 


Vocational 

Educator 


Suj 


icrvi sor 






High 

Effort 


Medium 

Effort 


Low 

Effort 


■ Ji 
: jvt 


Medium 

Effort 


Low 

Effort 


Hifch 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


H 

Li 




0% 




0% 




OX 


0% 


0% 


0% 


0% 


0% 


15% 


0% 




REVIEWS 


100% 




100% 




67% 


100% 


100% 


79% 


100% 


50% 


62% 


100% 




PRACTICAL 


0% 


0% 






0% 


0% 


0% 


0% 




0% 


0% 


0% 




, GUIDANCE 


80% 


80% 






100% 


0% 


75% 


50% 




63% 


100% 


100% 




PAPERS 






























0% 




0% 


ox 




0% 


0% 


0% 




17% 


0% 


0% 




BIBLIOGRAPHIES 


100% 




86% 


50% 




81% 


100% 


50% 




50% 


67% 


100% 







Researcher 


Instructional 
Resources Sped; 


alist 


Program 

Specialist 




P 


t incipal 








High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


l 

E i 


REVIEWS 


0% 

52% 


1% 

74% 


1Z 

72% 


8% 

75% 


0% 

70% 


0% 

56% 


5% 

58% 


OZ 

74Z 


2% 

70% 


0% 

100% 


0% 
6 ix 


0% 

100% 




PRACTICAL 

GUIDANCE 

PAPERS 


0% 

70% 


0% 

68% 


0% 

50% 


0% 

65% 


0% 

69% 


0% 

67% 


3% 

68% 


0Z 

88% 


0% 

89% 


0% 

78% 


ox 

100% 


0% 

100% 




BIBLIOGRAPHIES 


0% 

60% 


0% 

76% 


3% 

71% 


0% 

75% 


0% 

67% 


0% 

63% 


0% 

85% 


0% 

68% 


0% 

59% 


0% 

100% 


0% 

100% 


0% 

67% 





Secondary 

Teacher 


College 

Professor 


College 

Administrator 


Superintendent 


1 




High 


Medium 


Low 


High 

Effort 


Medium 


Low 


High 


Medium 


Low 


Effort 


Medium 


Low 






Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


Effort 


E 


REVIEWS 


0% 


7% 


0% 


2% 


i% 


1% 


0% 


0% 


0% 


0% 


0% 


0% 




100% 


86% 


70% 


70% 


73% 


75% 


93% 


77% 


79% 


92% 


78% 


91% 




PRACTICAL 


0% 


0% 


0% 


0% 


0% 


0% 


0% ■ 


0% 




0% 


0% 


0% 




GUIDANCE 

PAPERS 


75% 


100% 


87% 


60% 


80% 


79% 


71% 


75% 




77% 


50% 


100% 






0% 


0% 


8% 


3% 


0% 


2% 


0% 


0% 


0% 


0% 


0% 


0% 




BIBLIOGRAPHIES 


77% 


100% 


50% 


83% 


83% 


83% 


100% 


50% 


67% 


100% 


100% 


56% 





* 

In each cell, the top percent indicates unfavorable ratings 
percent favorable ratings? intermediate ratings are not rep' 
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TABLE VII-26. PERCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF "FORMAT" 

BY USER GROUP AND LEVEL OF EFFORT* 





Special 
i: ducat or 


Vocational 

Educator 


Supervisor 




Counselor 




rt 


High 
Ef fort 


Medium 

Effort 


Low 

Ef f ort 


Hifelt 

Effort 


Medium 
Ef fort 


Low 

Effort 


High 

Effort 


Medium 
Ef fort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 






0% 


0% 


0% 


0% 


0% 


0% 


15% 


0% 


• 0% 


0% 


7% 






67% 


100% 


100% 


79% 


100% 


50% 


62% 


100% 


25% 


75% 


791 






0% 


0% 


0% 


0% 




0% 


0% 


0% 


0% 


0% 


0% 


$ 




100% 


0% 


75% 


50% 




63% 


100% 


100% 


60% 


36% 


67% 


Vy 


0% 




0% 


0% 


0% 




17% 


0% 


0% 


0% 


0% 


0% 


fsi 


50% 




81% 


100% 


50% 




50% 


67% 


100% 


50% 


20% 


50% 





Instructional 
Resources Specie 


alist 


Program 

Specialist 




P 


rincipal 




Eli 

Ti 


ementary 

eacher 




rt 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


”P 

.* 


8% 


0% 


0% 


5% 


0% 


2% 


0% 


0% 


0% 


0% 




0% 




75% 


70% 


56% 


58% 


74% 


70% 


100% 


67% 


100% 


100% 




86% 


* / 


0% 


0% 


0% 


3% 


0% 


0% 


0% 


0% 


0% 


0% 


61 






65% 


69% 


67% 


68% 


88% 


89% 


78% 


100% 


100% 


33% 


75% 






0% 


0% 


0% 


0% 


0% 


0% 


0% 


0% 


0% 


0% 


ox 


0% 


z 


75% 


67% 


63% 


85% 


68% 


59% 


100% 


100% 


67% 


100% 


100% 


88% 





College 

Professor 


College 

Administrator 


Superintendent 


Other Administrative 
Position 


^rt 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


Hi, 

Ef: 


eh 

tort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Effort 




2% 


1% 


i% 


0% 


0% 


0% 




0% 


0% 


0% 


0% 


0% 


0% 


* 


70% 


73% 


75% 


93% 


77% 


79% 


92% 


78% 


91% 


0% 


50% 


75% 




0% 


0% 


0% 


0% 


0% 






0% 


0% 


0% 


0% 


0% 






60% 


80% 


79% 


71% 


75% 




77% 


50% 


100% 


76% 


100% 






3% 


0% 


2% 


0% 


0% 


0% 




0% 


0% 


0% 


0% 


0% 


0% 




83% 


83% 


83% 


100% 


50% 


67% 


100% 


100% 


56% 


50% 


50% 


100% 



•k 

In each cell, the top percent indicates unfavorable ratings and the bottom 
percent favorable ratings: intermediate ratings are not represented. 
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TABLE VII -27. PERCENTAGE OF FAVORABLE AND UNFAVOR 

BY USER GROUP AND LEVEL OF EFFORT* 





Reading 

Specialist 


Special 

Educator 


Vocational 

Educator 


Su] 


peryi 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


. . . . 

Low 

Effort 


High 
Ef fort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Med:' 

Eft 




OX 




02 




332 


0% 


0% 


OZ 


02 


OZ 


f 


REVIEWS 


751 




50% 




672 


672 


602 


89 Z 


802 


75% 


6 V 


PRACTICAL 
























GUIDANCE 


OX 


OX 


— 


— 


02 


02 


02 


OZ 


— 


0% 




PAPERS 


100 X 


80% 






882 


1002 


252 


50% 




75% 


6 


BIBLIOGRAPHIES 


ox 


— 


0% 


OX 


— 


OX 


0Z 


OZ 





OZ 






100X 




100% 


502 




812 


OZ 


100 Z 




83Z 


6 





Researcher 
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RCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF "RELEVANCE" 
' USER GROUP AND LEVEL OF EFFORT* 
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op percent indicates unfavorable ratings and the bottom 
ntings: intermediate ratings are not represented. 
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TABLE VII-28 . PERCENTAGE OF FAVORABLE AND 

BY USER GROUP AND LEVEL OF 
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iBLE VII-28 . PERCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF "NEED" 
BY USER GROUP AND LEVEL OF EFFORT* 
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TABLE VII-29. PERCENTAGE OF FAVORABLE AND UNFAVORABLE 

USEFULNESS" BY USER GROUP AND LEVEL OF E, 





Readirg 

Specialist 


Special 

Educator 


Vocational 

Educator 


Su 


pervisor 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


LOW 

Ef for 




02 




OZ 




OZ 


OZ 


OZ 


0% 


OZ 


0% 


OZ 


OZ 


REVIEWS 


75Z 




50Z 




67Z 


89Z 


60Z 


bv: 


80Z 


25% 


77Z 


88Z 


PRACTICAL 


OZ 


OZ 






OZ 


100Z 


OZ 


01 




0% 


OZ 


OZ 


GUIDANCE 


60Z 


40Z 






63Z 


OZ 


75Z 


0% 




75Z 


67Z 


33Z 


PAPERS 




























OZ 




OZ 


OZ 




OZ 


OZ 


ox 




OZ 


OZ 


OZ 


BIBLIOGRAPHIES 


100Z 




4 3Z 


100Z 




94Z 


OZ 


50S 




83Z 


33Z 


83Z 





Researcher 


Instructional 
Resources Specialist 


Program 

Specialist 


Principal 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effor 




2Z 


1Z 


OZ 


8Z 


OZ 


OZ 


5Z 


1Z 


3Z 


20Z 


OZ 


OZ 


REVIEWS 


60Z 


74Z 


5IZ 


42Z 


57Z 


5CZ 


63Z 


46Z 


56Z 


80Z 


5QZ 


78Z 


PRACTICAL 


4Z 


5Z 


OZ 


OZ 


OZ 


OZ 


3Z 


OZ 


5Z 


OZ 


OZ 


OZ 


GUIDANCE 

PAPERS 


59Z 


42Z 


OZ 


76Z 


54Z 


33Z 


61Z 


58Z 


53Z 


67Z 


OZ 


100Z 


BIBLIOGRAPHIES 


OZ 

80Z 


6Z 

71Z 


OZ 

65Z 


2Z 

50Z 


7Z 

60Z 


OZ 

70Z 


OZ 

88Z 


OZ 

67Z 


5Z 

67Z 


OZ 

BOX 


OZ 

50Z 


OZ 

67Z 





Secondary 

Teacher 


College 

Professor 


College 

Administrator 


Superintendent 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Effort 


High 

Effort 


Medium 

Effort 


Low 

Eff 




OZ 


OZ 


OZ 


OZ 


2Z 


2Z 


OZ 


OZ 


3Z 


OZ 


OZ 




REVIEWS 


100Z 


71Z 


40Z 


721 


57Z 


65Z 


47Z 


54Z 


62Z 


54Z 


44Z 


55 


PRACTICAL 


OZ 


OZ 


OZ 


n 


3Z 


4Z 


OZ 


OZ 




OZ 


OZ 




GUIDANCE 


75Z 


I00Z 


27Z 


33Z 


47Z 


58Z 


86Z 


75Z 




65Z 


OZ 




PAPERS 




























OZ 


OZ 


OZ 


OZ 


OZ 


OZ 


OZ 


OZ 


OZ 


OZ 


OZ 




BIBLIOGRAPHIES 


92% 


100Z 


67Z 


87Z 


75Z 


81Z 


100Z 


02 


89 Z 


67Z 


75Z 


l 7l 



ft 
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PERCENTAGE OF FAVORABLE AND UNFAVORABLE RATINGS OF ’ COMPARATIVE 
USEFULNESS” BY USER GROUP AN D LEVE L OF EFFORT* — 
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Effort 
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n each cell, the top percent indicates favorable ratings and the bottom 
percent favorable ratings; intermediate ratings are not represented. 




VII-71 

(page 72 blank) 

223 



Y ' COMPARISON OF EVALUATIONS BY "INTENDED AU DIENCES" 

The analysis by user groups for a single product (or issues of a series) 
produced some discriminating findings. The major issue to be addressed was 
one of how well an individual product was received by its intended audience. 

However, the actual question is a much broader one: how do user groups differ 

in their evaluations of a given product? 

Six individual products, plus combined evaluations of several issues from one 
series, were selected for inclusion in this analysis. These were selected 
from among approximately 20 products for which 25 or more evaluations were 
obtained. Final selections were made on the basis of two major criteria: 

1) that the size of each user group (or reasonable combination of similar 
user groups) be no less than 9; and 2) that the typology groups used did 
not disguise a more specific audience for which a particular product was 
intended so that findings might be misleading. 

A case in point helps to illustrate this last criterion. Two products, one 
from the ERIC Clearinghouse on Educational Media and Technology and the other 
from the Clearinghouse on Library and Information Sciences, easily met the 
first criterion. However, topics covered in the documents suggested that the 
more interested users for each document were either one or the other of two 
primary specialty groups that together comprise the typology group, Instruc- 
tional Resource Specialists. In other words, one product was more appropriate 
for media specialists and the other, for librarians. 

As alluded to earlier, this analysis could not simply compare one targeted 
audience with some other audience. In most cases, the single largest user 
group is compared to a miscellaneous group of ,f all other” respondents. 

Also, with the exception of some PREP reports, targeted audiences are not 
easily identified. Products are generally aimed at the research and practi- 
tioner communities in a subject area and/or educational level. Thus, to help 
the reader of this report judge just how "intended” or "peripheral” the user groups 
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are, the document representations used in the Evaluation Questionnaire survey 

precede their respective tables of findings (Exhibits VII-1 to 10 and Tables 

VII-31 to 37). 

Findings for each product are reported in percentages for six selected dimensions 
of quality and utility: relevance, need, coverage, up-to-dateness, format, 

and comparative usefulness. These percentages represent responses to the low- 
and high-choice points of each questionnaire item. For example, in the first 
ce ^ Table VII— 31, the percentages mean that in no cases (0 percent) did the 
Researchers judge this PREP report "not at all relevant" and that in 78 percent 
of the cases, they judged it "relevant." The missing percentage is the middle 
response point, so that in 22 percent of the cases, the document was judged to 
be "somewhat relevant." 

For this document, there is a spread of 51 percentage points between Program 
Specialists and Supervisors/Principals on the "need" item. The pattern of dif- 
ferences between user groups (drawn from Tables VII 31-37) is displayed in 
Table VII-30. Three levels of agreement were established to represent given 
spreads of percentage points (shown in parentheses) between user groups, for 
their high (i.e., most favorable) ratings on each item. A simple tally was 
performed on each document and the number in each column represents the total 
number of documents for which their user groups differed to that given degree. 

TABLE VII-30. SUMMARY TABULATION OF CASES OF DIFFERENCES BETWEEN USER GROUPS 



Quality/Utility Items 


Cases of 

General Agreement 
Between 
User Groups 
*■ (0-10 points) 


Cases of 

Some Disagreement 
Between 
User Groups 
(11-24 points) 


Cases of 
Signif icant 
Disagreement Between 
User Groups 
(25 points or more) 


Relevance 


1 


6 


1 


Need 


1 


4 


3 


Coverage 


0 


5 


3 


Up-to-dateness 


4 


V 

3 


1 


Format 


1 


3 


4 


Comparative Usefulness 


1 


3 


4 
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EXHIBIT VII-1. DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 7 






[PP016] 




[extract froa Introductory section] 



The kit contains 13 documents: 



In s national study of individualized in- 
structional programs --conducted by Jack V, 
Edling of the Oregon State System of Higher 
Education, Corvallis, for tho U.S. Office 
of Education--46 programs in 24 States were 
surveyed in depth. Ibis PREP kit reports 
on that study. 

The kit briefs school administrators and 
board mergers on the many approaches to 
individualizing instruction and tells of 
the experiences of those who have inaugu- 
rated such programs. Finally, it provides 
data updn which administrators and board 
members can make informed decisions con- 
cerning individualized instruction for their 
own schools or school districts. 



No. 16-A - 

No. 1G-B - 

No. 16-0 - 
No. 16-D - 
No. 16- E - 
No. 16-F - 
No. 16 -G - 

No. 16-H - 
No. 16-1 - 

No. 16-J - 
No. 16 -K - 

No. 16 -L - 

No. 16-M- 



Individualized Instruction: 

An Overview 

Objectives of Ind 4 vidualizcd 
Instruction 
Diagnostic Procedures 
Instructional Procedures 
Evaluative Procedures 
Student Progress Reports 
Evidence of Effects of 
Individualized Instruction 
Problems Encountered 
Recommendations on Implemen- 
tation Procedures 
Case Studies 

Materials for Individualizing 
Instruction 

Bibliography on Individualizing 
Instruction 

Current ERIC Documents on 
Individualizing Instruction 



In the eludy from which this report was derived, 46 school districts which hoe- 
made significant changes in their instructional programs were visited. In 40 
of them, the source of the change was directly attributable to an administra- 
tor, superintendent, assistant superintendent , principal, or a curriculum di- 
rector or his equivalent. In two instances, teachers played the key’ role. In 
the remaining four locations large-scale research and development projects 
were involved and had requested the school's cooperation. Even in those in- 
stances the school administrator had played a key vole in introducing the 
new instruction progri-i 

With reference to inti .mlizcd instruction as an innovation, the first 
question the adninistrut u r might ask is: "Do the new procedures now called 

individualized instruction merit adoption?" In the first phase, of this study 
nearly 1500 educators in key positions were contacted. Noe a single one raised 
the question, "Why should a school individualize its instruction program?" It 
would appear that there is almost universal acceptance of the principle that 
children differ, r.nd that these differences should be accommodated by differ- 
entiated learning experience*. Mott schools have avowed in their objectives 

[extract from section 16-1] 

1 
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Individualized Instruction. PREP Report $16 (N~76) 
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There is a clear indication that user groups differ considerably in such 
important areas as need, coverage, and comparative usefulness. 



Results for each of the remaining six products are contained in Tables VII-32 
through 37 . 
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EXHIBIT VII-2. DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 56 



[EM 001] 



[dr««m from Cover] 



* . 1 

I . • , 1 * ► 

tesfrusSSo:;;]! 

Instruction! 

ncicnals 

Materials 

Coolers 



/ 




i 



[sample entries] 

How Docs the Secondary School Library Become An \ 

Instructional Materials Center'’ Personnel, Program. j 

Materials, Housing 

Margaret Rogers, Oregon School Study Council, Lugcnc. j 

June 1968, Available as Document ED 027 049 from , 

EDRS, one fidie or 1 5 pages hardcopy j 

Objectives of this paper were { I ) to piovide j pra^tual j 

point of view, based on experience of library and judio. 
visual practitioners, for expanding secondary schunl \ 

library programs into instructional materials center 
programs as demanded by instructional programs inv^K 
mg flexible scheduling, inquiry , and independent study 
(2) to provide an annotated bibliography ot pertinent 
comment and illustrations from school administration, 
architecture, library , and audiovisual journals, books, and 
media; and (3) to make available to school administrators 
and planners selected sources on these topics The in- 
structional materials center, as defined for tlnsdis^uv 
sion, is a library with broader than traditional purpo>e, 
housing more diversified and extensive collections ot 
materials. It is staffed by individuals with varied general 
experience and specializations in library, curriculum, and 
media, who work supportivdy with staff and >tudcnt* 

This paper discusses personnel, program, materials, the 
indexing system, budget, and space allocations An anno- 
tated bibliography of 24 items is appended. 



Instructional Materials Centers -Selected Reading* 

Neviile ?. Pearson and Lucious Butler. 1969. Available 
from Burgess Publishing Co., 426 S. Sixth St,, Minneapolis 
Minn. 

Some 83 articles are presented m this 345-page paper 
bound book. Ail have appeared in one or another of 31 
periodicals in the last decade, with Audiovisual Instruc- 
tion by far the most heavily represented. Twenty^* of 
the articles are from AVI. The report literature, winch 
makes up most of the entries in this ERIC bibliography, 
of course is not represented in the Pearson and Butler 
book. But the articles ip Pearson and Butter are nowhere 
else so conveniently available. The book has sections 
devoted to the philosophy of the IMC, the IMC at dif- 
ferent educational levels, and operation , personnel and 
evaluation of IMCs. 



Don H, Coombs 

William J Paisley [drawn from Title Page] 

Michelle Timbie 
Len Schwarz 

Henry Ingle 



December 1969 
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DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 86 



EDUCATIONAL RESOURCES INFORMATION CENTER 

ERIC PRODUCTS 1969-1970 



[LI 008) 



A Bibliography of Information Analysis 
Publications of the ERIC Clearinghouses 
July 1969-June 1970 



[drawn from Title Page] 



Compiled by the 

ERIC CLEARINGHOUSE ON LIBRARY AND INFORMATION SCIENCES 
AMERICAN SOCIETY FOR INFORMATION SCIENCE 



[partial display] TABLE OF CONTENTS 


Page 


Introduction 


iv 


Statistical Summary 


Ti 


ERIC Products listed by Clearinghouse 


1 


Adult Education 


1 


Counseling & Personnel Services 


2 



ERIC Products [extract from Introduction] 

ERIC Products is an annual bibliography of those publica- 
tions of the ERIC Clearinghouses reflecting information analysis 
activities. It includes all substantial bibliographies, review 
papers, and state-of-the-art papers identified as ERIC publications; 
it does not include routine brochures, accession lists and short 
notes published in clearinghouse newsletters. This third issue of 
ERIC Products lists 366 publications of Fiscal Year 1970— July 
1969 through June 1970*. 



[•ample encriee] 



ADULT EDUCATION 



TO- ADULT BASIC EDUCATION- CURRENT INF0R- 
001 NATION SOURCES, NO. 2?. 89p- March 

1970. {Annotated Bibliography) 

ED 035 777 RIE June 1970. MF-10.50; 

HC-$S.55- 

The 261 referenced document* , 
aoatly from 1965, deal with survey*, 
planning, program description*, 
curriculum material*, clientele 
groups, teachers and administrators, 
teacher training, end recruitment. 



70- UNIVERSITY ADULT EDUCATION. (IN: 

008 HIOr'ZR ADULT EDUCATION IN THE UNITED 
STATES; THE CURRENT PICTURE, TRENDS 
AND ISSUES, Knowles, Malcom S. 
Washington, D.C.: American Council 

on Education, 1969. 12*p.) 50p. 

1969. (Annotated Bibliography) 

ED 03* 1*5 RIE April 1970. Not 
available. 

A 173-item eeleoted bibliography 
on university adult aducatlon. 
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ERIC Products 1969-1970 (N=41) 



All Others 


Instructional 

Resources 

Specialists 


USER GROUPS 
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EXHIBIT VII-4 . DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 136 




DEVELOPING CRITERION-REFERENCED TESTS 



{drawn from Cover and Title Page] 



Rex Jackson 



Test Development Division 
Educational Testing Service, Princeton, N.J. 



[June 1970] 



{abstract frog RESEARCH IN EDUCATION 1 

Present definitions of the criterion-referenced test are discussed, 
insufficiencies noted, and a new definition proposed. Some examples of 
criterion-referenced tests are examined and used to educe some general 
principles for the development of such tests. The utility of item form 
processes is assessed. It is suggested that the difficulty of objectively 
defining a test construction process is directly proportional to the com- 
plexity of the behavior the test is designed to assess. Problems and 
doubts with regard to the development of criterion-referenced tests for 
complex behavior domains -<re noted. In addition, some empirical methods 
for dealing with item an.Uy.'is, test reliability, and test validity 
difficulties are advanced 



Interest in criterion-referenced tests has risen in recent years as it 
has become increasingly clear that measures allowing only population-referenced 
interpretations do not provide the information that is needed in making certain 
types of decisions in education. Criterion-referenced measures have been con- 
sidered particularly desirable in areas where diagnostic information is needed, 
such as placement of individuals in programs of instruction or individual 
instruction, in formative evaluation of educational programs, and in evalua- 
tive assessment of individual or group achievement. {extrac£ from Conclusion] 
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EXHIBIT VII-5 . DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 93 



[RF 005} 



Reading: what can be measured? 



An IRA Research Fund Monograph from 
the ERIC/CRIER Reading Review Series 
[1969J 



Rtgir firr 

Indiana University 



[drawn from Title Page] 



[extract from Chapter 1] 



Measurement in reading: general perspectives 



This monograph organises and describes the research litera- 
turc on measurement and evaluation in rending- The review of 
the research is by no means exhaustive and while the major 
controversies in the field have been outlined, no attempt has 
been made to resolve them (although, in some instances, direc- 
tions for possible solutions have been offered) The mono- 
graph is intended to serve as a guide to the researcher in point- 
ing out both what is known and what is not known sn measure- 
ment and evaluation in reading as well as to delineate those 
treas which need further research. The monograph also pro- 
vides guidelines for the classroom application of research and 
explains how the teacher can and should use the wide array of 
measuring devices available. A guide to tests and measuring 
devices in reading has been included as a companion piece to 
the monograph. In it are listed reading tests currently in print. 
Information about the grade levels at which the test* is appro- 
priate, the kinds of sub-tests included within the test, the num- 
ber of forms the test has, and the amount of time needed for ad- 
ministration are included In addition, the Guide makes it possi- 
ble for the teacher or researcher to obtain further information 
about any particular test either by writing to publishers (whose 
addresses appear in the Guide), by checking the reviews in 
Buros ( 1968) Reading Tests and Reviews, or by consulting re- 
search which has used these tests, easily available through the 
published journal literature which is described in documents 
from the ERIC/CRIER system. 



The major theme of the monograph is the use of tests in 
providing information about students 1 reading achievement. 
Such information is necessary to the teacher in setting instruc- 
tional goals and in helping Modems to develop their reading 
skills Thus, the first step in any discussion of testing and eval- 
uation in reading is to define those skills which are essentia! to 
the rc.tdmc act Once this is done, then it js possible to con- 
sider whether reading tests accurately assess reading behavior 
Can what they measure serve as a basis for organizing class- 
room instruction? 

Contents ffuii 



Introduction 

Measurement in reading: general perspectives/ 1 

Skids underlying reading ability Variables affecting reading 
performance the student's background Variable* affecting 
reading performance the reading program In conclusion 

Problem! in measuring reading sub-sktfis/2 

Reading vocabulary Speed of rCad.ng Reading compre- 
henwon Rate of wom r r C hen s ,on What can be measured^ 

Methods for assessing reading achicvcmcnf/3 

Standardized tests Informal measurement of reading 
A norc to the practitioner 



vii 

I 



33 

SO 



Assessing growth/4 

Difficulties m assessing growth Procedures for assessing 

change Measuring growth i*o unique ctoes 



Measurement of reading-related var«abIes/5 

Relation between intelligence and reading The use of other 
psychological measures in assessing reading ub.hcy jh e usC 
of physiological measures to estimate reading capacity A 

note to the practitioner 



Summary: lest uses and research nccds/6 


212 


Glossary 


219 


Guide to tests and measuring instruments in reading 


225 


lodes to Reading Test, 3n d R ev ie„, and 




Mental Measurement Yearbooks 


271 


Index to published research literature in reading 


284 
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TABLE VI 1-34. COMPARISON OF EVALUATIONS BY USER GROUPS FOR DOCUMENT #136. 




Reading: What Can Be Measured ? (N=41) 



EXHIBIT VII-6 . DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 118 






A READER'S GUIDE TO THE 
COMPREHENSIVE MODELS FOR PREPARING 



[SP 003] 



ELEMENTARY TEACHERS 



Published by th« 

ERIC CLEARINGHOUSE ON TICKER EDUCATION 



end the 

AMERICAN ASSOCIATION OF COLLEGES FOR TEACHER EDUCATION 
Oae Dupont Circle 
Washington, D. C. /December 1969 



Edited by Joel L. Burdin 
end Kaliopee Lanzillotti 
ERIC Clearinghouse on Teacher Education 

[drawn from Title Page] 

[full display] TABLE OF CONTENTS PAGE 

FOREWORD 

INTRODUCTION 

HOW TO USE THIS GUIDE xiii 

ABOUT ERIC 



GUIDES TO THE MODELS i 

Florida State University 3 

Michigan State University ,23 

Northwest Regional Educational Laboratory 51 

Syracuse University ! . 85 

Teachers College , Columbia University. ....... 105 

The University of Georgia. . ^59 

The University of Toledo ..... ........ 197 

University of Massachusetts 211 

University of Pittsburgh 233 

SECOND-DAY INTERACTION OF THE WRITERS’ CROUP 277 

index 301 



[extract] HOW TO USE THIS GUIDE 

This publication has three main sect ions--guides to each of the nine 
models, a section on the second-day interaction, and an index which 
provides cross-references. 

The guides all have this general outline: overview, program goals 

and rationale, selection procedures, professional preservice component, 
relationship of professional component to academic component, inservice 
component, faculty requirements and staff utilisation, evaluation com- 
ponent, program management, and summary. The Teachers College guide, 
which was not written at the conference, is the only one with a 
different outline. 
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[extracts from pn*e i of foment] 



EXHIBIT VII-7 . 



DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 74 



[JCOllj 




Volume 4. Number 3 



JUNIOR COLLEGE RESEARCH REVIEW 

♦ November 1969 



ruWi*h«d by th. Am«i+e*n A*tOCl»«ton of Junior Cotl»i*» 



COLLEGE-COMMUNITY RELATIONS 



Barton R. Htmchrr 
Regional Education Laboratory 
for the Caroltms and Virginia 

Thomas M . Hatfield 

John Tyler Community College. 



The emergence of the concept of the public junior 
college as a community college has been, in the view 
of B Lamar Johnson, the most important junior col- 
lege development cf the past 40 years Greater em- 
phasis is being placed on the "community" aspect of 
community colleges as the tremenduus growth of the 
past decade continues into the 1970's. Through the 
provision of community' service programs and curricula { 
adapted to the needs of the populace, the community < 
college earns its name. It is obvious, however, that not { 
all community colleges are, in the strict sense, con\- 
munity" colleges. Some critics have even suggested 
that the name "community college* is, in far too many 



Virginia 

instances, a shibboleth \ concerted effort toward the 
promotion of closer college-community relations is the 
key to making its name truly descriptive of the role 
ascribed to it. 

This issue of the Junior College Research Revieu ex- 
amines documents that focus specifically on the issue 
of college-community relations. They were selected 
from material received and processed by the ERIC 
Clearinghouse for Junior Colleges. All have hern in- 
dexed and abstracted in Research in Education. Copies 
of the reports, both in hard copy and microfiche, are 
available from ERIC Document Reproduction Service 



Review 

The documents reviewed cover a variety of topics 
bearing on college-community relations: programs de- 
signed to serve community needs, public relations, the 
college image, advisory committees, and community 
opinion. This Review wilt not undertake an examination 
of the community service function of the junior college, 
except where it is touched on within the context of the 
broad topics listed above. 



[sample entries] 



BIBLIOGRAPHY 



:D 013 047 t J 

\uide to Public Relation* for Junior Colleges . by Alexander 
I Streloff. South-Western Publishing Co., Burlingame, 
California, March 1961. 43 p. (MF4Q.25, HC-12.25) 



• ED 014 985 

Report of Research Protect to Determine Effect of Stoss 
Circulation of Macomb County Community College Stu- 
dent Newspaper on Public Inters of the College, by Miles 
Meyarson. Warren, Michigan. Macomb County Community 
Collage, August 14, 1967. 9 p (MF-fO.25. HC.f0.S5) 
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EXHIBIT VII-8 . DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 76 




f.TC 0161 

Vo 4, Number 6 



JUNIOR COLLEGE RESEARCH REVIEW 



February 1970 



Published by tht Amtncm Association of Junior Cotf«*g«s 



[from page 1 of document] 





NEEDED: RATIONAL CURRICULUM PUNNING 

Edgar A. Quimby, Page 2 



NEEDED: 

RATIONAL CURRICULUM PUNNING 

Edgar A. Quimby 

Inititule for Development of 
Educational Activities, Inc. 

(California} 

Junior college* interested in rational curriculum planning 
will have to develop molt of their own tools, because there li 
precious little writing on junior college curriculum planning 
in the literature In fact, the latest substantive discussion of 
curricular issues in the two-year college. B Lamar Johnson’s 
General Education in Action [7], is now nearly twenty years 
old However regrettable the tack of curricular development 
tools may be. it is understandable for two important reasons 
On the one hand, ever-increasing attention has been devoted 
in recent years to an apparently widely recognized need for 
improving instruction in two-year colleges, and many of the 
most thought-provoking writings in the past decade have been 
concerned with innovative programs and improved instme- 

[extract from firat paper] 



THE MINI-COLLEGE REVISITED 

Afvin T. Beart, Vernon L. Hendrix, Page 5 



BUCK STUDIES 

John Lombardi, Page 7 



A STAFFING RATIONALE 
FOR CURRICULUM DEVELOPMENT 
IN THE COMMUNITY COLLEGE 
George H. Voegel, Page 10 



INTERCAMPUS CURRICULUM 
COORDINATION ^ AN URBAN 
COMMUNITY COL EGE SYSTEM 



James 



I. Cox . Page 14 
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EXHIBIT VII-9. DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 77 




[JC01'9] 

Volume 5, Number 2 



JUNIOR COLLEGE RESEARCH REVIEW 



October 1970 



Fubii*h»d by int Amtocan Mioci*tiopi of iumor CcHfget [drawn from page l of 
document] 



COOPERATIVE WORK-EXPERIENCE EDUCATION PROGRAMS IN JUNIOR COLLEGES 

Murcia A. Boyer 
Information Anahjst 



Occupations within business nd industry art* more, 
specialtfed and diversified than ever before At the 
same time, an cver-incrcasing share of the responsi- 
bility for providing post-secondary education in this 
country is ncing allotted to the junior college Unfor- 
tunately, many junior college* find themselves in thi 4 
almost universal quandarv of lacking funds, facilities, 
and manpower. To meet their educational responsibili- 
ties, junior college have had to adapt huth tru ir cur- 
ricula and teaching methods to make student learning 
experiences compatible with, and relevant to, the needs 
of ousiness and industry. One innovation that ha* grown 
In application and scope is the cooperative work-expe- 
rience education program, which combines course work 
with directly related employment. These programs are 
distinguished from other types of student employment, 
which may be only casual in nature, hy the fact that in 
them the students employment is an integral part of 
his college program and is supervised and evaluated 



cooperatively by a college coordinator or instructor and 
his employer. 

Those interested in exploring the possibilities of cxi 
operative work experience programs for their college 
may ask the following questions: What are their spe 
cific benefits? What type* of curricula lend themselves 
to this cooperative arrangement* How arc the pro 
grams administered"* What problem areas can he 
anticipated and possibly avoided, through cart ful 
planning* 

This issue of the Junior College Research Rcvicu 
addresses these aspects of cooperative work experience 
education programs. Documents cited in this review 
were selected from materials received and processed hy 
the EPIC Clearinghouse for Junior Colleges All docu- 
ments listed in the bibliography have been announced 
in Re tearch in Education and may he obtained from 
EDRS, as explained on page 4. 
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EXHIBIT VII-10. DOCUMENT REPRESENTATION (IN REDUCED FORM) FOR DOCUMENT 78 
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[extract from page 1 of document] 

Volume 5, Number 3 



JUNIOR COLLEGE RESEARCH REVIEW 



November 1970 



Pub!tftft«d by the American Association of Junior CoHefes 



OCCUPMTIONALLV 

ORIENTED STUDENTS* 

K. Patricia Cross 



For the past twenty years, this nation has been working 
toward an explicit goal of universal higher education. The 
concept has found ready acceptance by both political par- 
ties and by four American Presidents since Truman's Com- 
mission on Higher Education proclaimed in 1947 that “At 
least 49* of our population has the mental ability to com- 
plete fourteen years of schooling with a curriculum of 
general and vocational studies that should lead either to 
gainful employment or to further study at a more ad- 
vanced level** T n 1947, when only one-fourth of the 18- 
and I9 ; year olds were in college, the proposal was her- 
alded as a bold ideal From our perspective now, it seems 
quite modest. We have already surpassed the goal they 
envisioned and, by 1980, two-thirds of the college-agt* 
/uuth will be in college. We are no longer concerned with 
whether students are ready for higher education, but 
rather with whether higher education is ready for them. 

Not long ago, higher education addressed itself to a 
limited segment of trie population. The academic mode! 
served reasonably well, and each level of education was 
judged by how well it prepared students for the next 
level. Past school grades were, and still are, the best pre- 
dictors of future grades. Admissions tests did, and still do. 
an adequate job of predicting success in college, if success 
is defined along traditional academic lines. Our national 
commitment to universal post-secondary education, how- 
ever, ha: brought us face-to-face with the reality that we 
must educate youth tor life in a society where knowledge 
is exploding, semi-skilled and unskilled jobs are disappear- 
ing, and most of the population will have to run just to 
stay in place with the demands for new skills. In Venn's 
(lv64) colorful words, technology has placed education 
squarely between man and his work" (21). 



also has a past to overcome. Because of our narrow joi 
demic definition of higher education, occupational e<Iut j 
tion has never been quite “academically rt*>pc a i table," n*>r 
have the young people in it been considered ‘’talented " 
Occupational education has all too often been thougM *4 
in negative terms, i.e ., students take occupational courses 
not because of what they can do, but because of w h.it 
they can't do. 

Certainly ;tudents in the occupational curmtil.i .if tin- 
community colleges today are an early taste ut the de- 
mands that universal higher education will make on edu 
cational innovators. To give up the educational techniques 
that have not worked and to find new ones that will is the 
challenge, and it will take much better understand inn 
than w r e now have of the characteristics of the student 
who is new to the ranks of higher education. Although 
the research is scanty, a synthesis of scattered bits of data 
may help to construct a tentative description »<f the ^har 
acteristics of the occupationally-oriented studcrif 

Although it simplifies things to speak ol both students 
enrolled in the technical degree programs and those in the 
vocational non-degree curricula of the community college 
as occupationally-oriented, it should be noted that mans 
of them say that they hope to transfer to a four-year col- 
lege. 



•Prepared for a two-day conference jointly sponsored hy the 
American Educational ruhlahe-r* Institute and thf Ameru 4» 
Association of Junior College* on Occupational- Oriented Pn 
grams in Two-Year Colleges, in Miami, Florida, Decertify r r > 
1969. 



(sample entries] 



-! 
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TABLE VII-36. COMPARISON OF EVALUATIONS BY USER GROUPS FOR DOCUMENT #118 
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Four issues of Junior College Research Review Series (N»lll) 



G. 



EVALUATION OF PRODUCTS IN THE "DISADVANTAGED” SUBJECT AREA 



A specific subject analysis was performed on products covering topics related 

•k 

to the area of the Disadvantaged. This area was selected for two reasons: 

1) it is a clear example of a USOE priority area and 2) a large pool of 
pertinent documents could be identified, thereby ensuring a sufficient 
number of evaluations for analysis. 

Twenty-five products are included in this analysis. They were identified on 
the basis of titles and descriptors ( from Research In Education ) , and no 
further selection criteria were applied. Specific products are identified below 
by their document code numbers (see Appendix A for full bibliographic citations) . 
This total group represents several PREP reports, one EMC bibliography, and 
products of seven ERIC clearinghouses . 

ews Practical Guidance Bibliographies 



(N=7) 


Papers 




(N=8) 


(N«10) 


34 


2 


10 


35 


3 


28 


36 


6 


29 


37 


14 


30 


38 


22 


31 


98 


24 


32 


100 


25 


33 




70 


95 






116 






130 



The results of this analysis are reported in Table VII-38. The first (or 
top) mean in each cell is the Disadvantaged 11 mean. Reference means (i.e., 



References were made in Sections B, C, and D of this chapter to findings 
from an analysis by general subject area (i.e., for products in the areas 
of Instructional Content, Educational Administration and Services, Special 
and Other Educational Groups, and Higher Education). Master tables for this 
analysis are contained in the concluding section of this chapter 
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grand means for each product type) are provided below in parentheses so that two 
types of comparisons can be made: 1) between Disadvantaged means and Reference 

k 

means for each item; and 2) among Disadvantaged means for each item across 
product types. 



For example, in the first cell, the 2.70 is the Disadvantaged mean for "rele- 
•> vance M evaluations of Reviews; 2.72 is the Reference mean of "relevance" 
evaluations for all Reviews. In this comparison, there are only two 
instances of any significant differences at the .01 level: 1) on the ’'need’ 1 

item, the Disadvantaged-group respondents indicated that bibliographies filled a 
significantly greater need than was indicated by the overall population of 
respondents; and 2) the specific population was much more critical of the format 
of bibliographies in the Disadvantaged area, with a relatively low mean of 
2.35. Although not statistically significant, the Disadvantaged means for 
each item were lower than the norm in over 50 perc.ent of the cases. 



In comparing only Disadvantaged means by product type, Practical Guidance 
Papers were lowest of the three types in 50 percent of the cases. The need for 
Reviews was lowest and contrasts significantly with the need for Bibliograpnies. 

* m 

Bibliographies represent the low Disadvantaged means for "up-to-dateness" and 
"format." 

Low means for Practical Guidance Papers are evident in two other areas (not shown 
in the Table) : 

Reviews Practical Guidance Papers 

Adequacy of Discussion 2.41 2.15 

Clarity of Writing 2.56 2.39 



The difference in the "adequacy of discussion" is a significant one. 



From Readers’ impact data. Reviews in the Disadvantaged area were more used in 
making decisions than were Practical Guidance Papers, with 30 percent in contrast 



k 

For a comparison between some of the respondents and the remainder of the 
respondents, it is desirable to recompute the Reference means to exclude the 
Disadvantaged group. This was done prior to computing the tests for significance 
of difference referred to in this section. 
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to 17 percent. In other ways, the two product types were fairly comparable. 

For example, the response "applied the information in my work" was marked in 
68 percent of the cases for Reviews and 67 percent, for Practical Guidance 
Papers. Bibliographies in this subject area were used most frequently (67 
percent) to identify relevant literature by topics; and secondly (65 percent) for 
seeing what kind of new work is being reported. This order of principal use is 
the same as that in the general population of respondents. 

Although there were not a sufficient number of products in other USOE priority 
areas (e.g. , in the reading area, there were only five products in the sample) 
to pursue this type of analysis, the results would suggest that such an 
analysis can produce useful information of differences in needs and preferences 
among users for different subject areas. 
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EVALUATION BY LEVEL OF VISIBILITY 



The visibility-index groupings of products for analysis were most useful in 
assessing the relation of "high visibility products*' witn reader familiarity 

(See Chapter V). There is, however, no strong suggestion of a pattern in the 
relation of visibility to quality, utility, and impact ratings on products. 

A selected display of these findings is shown in Table VII-39. (Master reference 
tables are provided in the next section.) Only with Reviews does a high 
visibility group differ significantly in several cases from the low-visibility 
product ratings. Since these differences are significant, we might speculate 
that the joint development and dissemination of Reviews between clearinghouses 
and other agencies (e.g., professional associations) have some effect on this 
apparent relation of quality and high visibility. These products may 
receive special attention in preparation because their dissemination strategies 
are planned and possibly more extensive. As one Specialist pointed out in a 
lengthy letter noting the cost/quality dilemma in the ERIC system, the 
pressure of knowing that thousands will see a paper is one of several elements 
that can contribute to a quality product. 
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TABLE VII-39. EVALUATIONS ON SELECTED QUALITY DIMENSIONS BY LEVELS OF 

VISIBILITY 





QUALITY (of Reviews) 






Low 

Visibility 

[N=342] 


Medium 

Visibility 

[N«314] 


High 

Visibility 

[N=*301] 


Reference* 

Mean/Percent 

[N=9571 


Coverage 


2.47 


2.50 


>.54 


(2.50) 


Up-to-dateness 


2.77 


2.82 


2.84 


(2.81) 


Organization 


2.30 


2.29 


2.36 


(2.31) 


Writing 


2.48 


2.49 


2.57 


(2.51) 


Format 


2.69 


2.71 


2.77 


(2.72) 


Discussion 


2.25 


2.32 


2.39 


(2.32) 



, 




QUALITY (of Practical Guidance Papers) 




Low 

Visibility 

[N-77] 


Medium 
Visibility 
f N— 126 1 


High 

Visibility 
fN=2S!T 


Reference 
Mean/Percent 
[N= c 4 56 1 


Coverage 


2.40 


2.35 


2.49 


(2.43) 


Up-to-dateness 


2.79 


2.76 


2.77 


( 2.77) 


Organization 


2.29 


2. S3 


2.34 


(2.33) 


Writing 


2.55 


2.52 


2.53 


(2.53) 


Format 


2.74 


2.76 


2.73 


(2.74) 


Discussion 


2.30 


2.25 


2.32 


(2.30) 





QUALITY (of Bibliographies) 


- 




Low 

Visibility 

TN=2441 


Medium 

Visibility 

fN*=2301 


High 

Visibility 
fN“27 1 


Reference 

Mea ?6£S5if t 


f “■ 

Coverage 


2.50 


2.48 


2.44 


(2.49) 


Up-to-dateness 


2.80 


2.74 


2.89 


(2.78) 


Organization 


2.23 


2.24 


2.07 


(2.23) 


Format 


2.73 


2.73 


2.59 


(2.72) 


Textual material 


2.51 


2.43 


2.41 


(2.47) 
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SUMMARY ANALYSES OF SELECTED VARIABLES 



This final section contains master reference tables for the analyses discussed 
in this chapter. They are presented in the following order: 

READER DATA 

Overall Evaluation of Reviews, Practical Guidance Papers, and 
Bibliographies, and each by: 

• Level of Effort 

• Level of Visibility 

• Subject Area 

• User Group 

SPECIALISTS * DATA 

Overall Evaluations of Reviews, Practical Guidance Papers, and 
Bibliographies 

The concluding Table in this section can be used with any of these tables by 
the reader wishing to compute the approximate significance of difference 
between any two means. Instructions for conducting these tests are also provided. 



TABLE VII-40. READERS’ OVERALL EVALUATION OF REVIEWS 



READER EVALUATIONS 


(N-957) 










QUALITY 


UTILITY 








Mean 




Mean 


Coverage 




2.50 


Relevance 


2.72 


Up-to-dateness 




2.81 


Need 


2.33 


Organization 




2.31 


Comparative usefulness 


2.58 


Writing 




2.51 


Purpose of use: 




Format 




2.72 


Obtain overview 


2.63 


Discussion 




2.32 


Look up facts 


2.20 






Percentage 


Identify individuals 


2.13 
















Identify relevant 


2.36 


Length: 




- 


literature 


About right 




82% 


Update knowledge 


2AL 


Too long 




4% 


Obtain new 


2.14 


Too short 




10% 


knowledge 






IMPACT 










Percentage 




Used to make decision 


19% 




Applied in my work 




69% 




Used to give advice 


42% 




Examined other 


documents 


32% 




Consulted with 


author (s) or others 


8% 




Passed document 


: on 


to colleague(s) 


46% 
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TABLE VI 1-41 



READERS ? EVALUATION OF REVIEWS BY LEVELS OF EFFORT 





QUALITY 




Low Effort 
[N-349] 


Medium 

Effort 

[N-406] 


High Effort 

{N— 202] 


Reference 

Mean/Percent 

[N-«57] 


Coverage 


2.50 


2.50 


2.50 


(2.50) 


Up- to-datenes* 


2.79 


2.81 


2.83 


(2.81) 


Organization 


2.34 


2.31 


2.28 


(2.31) 


Writing 


2.56 


2.50 


2.45 


(2.51) 


Format 


2.75 


2.72 


2.67 : 


(2.72) 


Discussion 


2.34 


2.29 


2.32 


(2.32) 


Length: 

About right 


83% 


83% 


82% 


(82%) 


Too long 


2% 


4% 


6% 


( 4%) 


Too short 


12% 


10% 


8% 


(10%) 



UTILITY 



Relevance 


2.69 


2.76 


2.70 


(2.72) 


Need 


2.28 


2.37 


2.34 


(2.33) 


Comparative usefulness 


2.57 


2.58 


2.58 


(2.58) 


Purpose of use: 
Obtain overview 


2.61 


2.63 


2.65 


(2.63) 


Look up facts 


2.15 


2.22 


2.23 


(2.20) 


Identify individuals 


2.13 


2.14 


2.11 


(2.13) 


Identify relevant 
literature 


2.31 


2.37 


2.43 


(2.36) 


Update knowledge 


2.46 


2.48 


2.46 


(2.47) 


Obtain new knowledge 


2.14 


2.15 


2.12 


(2.14) 



IMPACT 



Used to make decision 


15% 


19% 


24% 


(19%) 


Applied in my work 


66% 


72% 


66% 


(69%) 


Used to give advice 


42% 


42% 


45% 


(42%) 


Examined other documents 


28% 


34% 


32% 


(32%) 


Consulted with author(s) 
or others 


6% 


10% 


7% 


( 8%) 


Passed document on to 
colleague (s) 


44% 


44% 


52% 


(46%) 




A. 



V. 
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TABLE VI 1-42 



READERS’ EVALUATION OF REVIEWS BY LEVEL OF VISIBILITY 





QUALITY 








Low 

Visibility 

[N-342] 


Medium 

Visibility 

TN-314] 


High 

Visibility 

fN-301] 


Reference 

Mean/Percent 

[N-957J 


Coverage 


2.47 


2.50 


2.54 


(2.50) 


Up-to-dateness 


2.77 


2.82 


2.84 


(2.81) 


Organization 


2.30 


2.29 


2.36 


(2.31) 


Writing 


2.48 


2.49 


2.57 


(2.51) 


Foraat 


2.69 


2.71 


2.77 


(2.72) 


Discussion 


2.25 


2.32 


2.39 


(2.32) 


Length: 








(82%) 


About right 


84% 


81% 


82% 


Too long 


3% 


5% 


4% 


( 4%) 


Too short 


9% 


10% 


12% 


(10%) 



UTILITY 



Relevance 


2.71 


2.72 


2.73 


(2.12} 


Need 


2.32 


2.35 


2.32 


(2.33) 


Comparative usefulness 


2.54 


2.58 


2.62 


(2.58) 


Purpose of use: 
Obtain overview 


2.61 


2.61 


2.66 


(2.63) 


Look up facts 


2.13 


2.24 


2.24 


(2.20) 


Identify individuals 


2.11 


2.17 


2.11 


(2.13) 


Identify relevant 
literature 


2.34 


2.36 


2.38 


(2.36) 


Update knowledge 


2.43 


2.48 


2.50 


(2.47) 


Obtain new knowledge 


2.13 


2.17 


2.13 


(2.14) 



IMPACT 



Used to make decision 


. 16% 


23% 


18% 


(19%) 


Applied in my work 


68% 


68% 


70% 


(69%) 


Used to give advice 


38% 


43% 


47% 


(42%) 


Examined other documents 


29% 


34% 


32% 


(32%) 


Consulted with author (s) 
or others 


6% 


12% 


6% 


( 8%) 


Passed document on to 
colleague(s) 


44% 


44% 


50% 


(46%) 
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TABLE VI 1-43 



READERS’ EVALUATIONS OF REVIEWS BY GENERAL SUBJECT AREA 



l 


QUALITY 




i 

Instr . 
Content 
[N-301] 


Ed. Adm./ 
Services 
[N-200] 


Special/ 
Other Educ. 
Groups 
[N“174] 


Higher 

Educ. 

fN-2821 


J Reference 
1 Mean/Percent 
TN=957 1 


Coverage 


2.56 


2.51 


2.48 


2.44 (2.50) 


Up-to-dateness 


2.84 


2.80 


2.74 


2.82 


(2.81) 


Organisation 


2.33 


2.29 


2.33 


2.30 


(2.31) 


Writing 


2.52 


2.49 


2.55 


2.50 


(2.51) 


Format 


2.71 


2.74 


2.69 


2. 74 


(7. 72) 


Discussion 


2.34 


2.29 


2.34 


2.29 


(2.32) 


Length: 

About right 


85% 


832 


83% 


78% 


(82%) 


Too long 


4% 


42 


n 


4% 


( 4%) 


Too short 


8% 


82 


n 


16% 

1 


(10%) 


UTILITY 




Relevance 


2.77 


2.74 


2.68 


2.69 


(2.72) 


Need 


2.40 


2.34 


2.28 


2.28 


(2.33) 


Comparative usefulness 


2.65 


2.54 


2.55 


2.55 


(2.58) 


Purpose of use: 
Obtain overview 


2.66 


2.62 


2.54 


2.65 


(2.63) 


Look up facta 


2.21 


2.22 


2.18 


2.18 


(2.20) 


Identify Individuals 


2.15 


2.12 


2.11 


2.13 


(2.13) 


Identify relevant 
literature 


2.40 


2.33 


2. 30 


2.38 


(2.36) 


Update knowledge 


2.50 


2.46 


2.51 


2.42 


(2.47) 


Obtain new knowledge 


2.16 


2.19 


2.09 


2.11 


(2.14) 




IMPACT 






Used to make decision 


242 


21% 


13% 


15% 


(19%) 


Applied in ay work 


702 


64% 


66% 


72% 


(69%) 


Used to give advice 


472 


39% 


40% 


41% 


(42%) 


Examined other documents 


352 


33% 


25% 


32% 


(32%) 


Consulted with author (s) 
or others 


62 


10% 


7% 


9% 


( 8%) 


Passed document on to 
colleague(s) 


452 


45% 


48% 


46% 


(46%) 
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TABLE VTI-44 . READERS’ EVALUATION OF REVIEWS BY GENE 



QUALITY 





Xi-aiiint; 

• 

[N=d] 


Spec . 
!\!u« . 
[N=12j 


Voe . 
i\im’ . 
[ N = 2 9 ] 


Super - 
v i 

IN-25] 


Cmm- 
■n- lor 
[ N= U) J 


K & l) 

St .if !' 
|N-19l f 


1 ns tr . 
Ur s . Sp 
[;^5M | 


Spn* . 

I N“ 151] 


I'r i n- 
r ! jvi l 
! N - 2 ( t J 


El on: . 
Tr.ii’h . 
(N--9| 


Sts . 
If.ii h . 

j N- 2b | 


Uil 1 . "" 
Prot . 

I N=2 52 1 


" r.< . i \ 

A« i in i 
t N=- 


Coverage 


2. )i 


2 . 50 


2.^5 


2 . 68 


2.4 3 


2.42 


2.5* 


2.48 


2.60 


2 . 3 3 


2.81 


2.54 


2.41 


Up-to-dateness 


: .Hi 


2.67 


2.79 


2.88 


2.7 7 


7.79 


2 . 88 


2.7 6 


3.90 


7.b7 


2.81 


2.83 


2.85 


Organization 


2. 50 


2.17 


2.21 


2.28 


2.20 


2.32 


2.2S 


2.36 


2.25 


2 . 22 


2.54 


2.31 


2.27 


Writing 


2.^3 


2.50 


2.34 


2.56 


2.47 


2.53 


2.47 


2.59 


2.75 


2 . 11 


2.69 


2.47 


2.47 


Format 


3.00 


2.92 


2.86 


2.64 


2.67 


2.66 


2.66 


2.70 


2.90 


2,. 89 


2.7? 


2.72 


2.81 


Discussion 


2.33 


2.17 


2* 21 


2.40 


2.27 


2.31 


2.21 


2.36 


2.40 


2.11 


2.65 


2.32 


2.28 


Length: 

About right 


83% 


75% 


90% 


84% 


93% 


80% 


84% 


79% 


80% 


67% 


85% 


85% 


82% 


Too long 


17% 


8% 


0% 


4% 


0% 


4% 


5% 


- 3% 


5% 


22% 


4% 


3% 


6% 


Too short 


* 0% 


8% 


7% 


12% 


3% 


13% 


9% 


13% 


10% 


11% 


8% 


9% 


KK 



UTILITY 



Relevance 


2.67 


2.58 


2.83 


2.76 


2.60 


2.67 


2.57 


2.74 


2.75 


2.67 


2.88 


2.79 


| 2.71 


Heed 


2.83 


2.25 


2.31 


2.44 


2.30 


2.25 


2.10 


2.33 


2.15 


2.22 


2.54 


2.44 


2.30 


Comparative usefulness 


2.67 


2.83 


2.59 


2.72 


2.40 


2.61 


2.50 


2.50 


2.65 


2.44 


2.62 


2.62 


2.54 


Purpose c use: 
Obtain overview 


3.00 


2.25 


2.55 


2.76 


2.50 


2.62 


2.40 


2.62 


2.80 


2.56 


2.88 


2.65 


2.68 


Look up facts 


2.50 


2.17 


2.24 


2.16 


2.23 


2.16 


2.09 


2 , 26 


2.40 


2.22 


2.27 


2.21 


2.15 


Identify individuals 


2.17 


2.08 


2.14 


2.08 


2.03 


2.18 


2.07 


2.17 


2.10 


2.00 


2.15 


2.14 


2.06 


Identify relevant 
literature 


2.33 


2.42 


2.28 


2.20 


2.23 


2.42 


2.43 


2.34 


2.35 


2.22 


2.50 


2.38 


2.28 


Update knowledge 


2.83 


2.58 


2.38 


2.68 


2.57 


2.41 


2.33 


2.4 7 


2.65 


2.33 


2.77 


2.47 


2.49 


Obtain new knowledge 


2.17 


2.17 


1.97 


2.20 

j 


2.03 


2.14 


2.19 


2.17 


2.30 


^2.33 


2.08 


2.14 


2.09 



IMPACT 



Used to make decision 


33% 


33% 


3% 


28% 


23% 


15% 


22% 


18% 


40% 


0% 


38% 


17% 


22% 


Applied in my work 


83% 


75% 


66% 


68% 


73% 


68% 


36% 


73% 


7 5% 


56% 


GO 


71% 


71% 


Used to give advice 


67% 


50% 


34% 


60% 


33% 


3 5% 


33% 


46% 


45% 


0% 


42% 


47% 


48% 


Examined other documents 


50% 


33% 


52% 


16% 


20% 


37% 


24% 


36% 


20% 


22 % 


50% 


31% 


24% 


Consulted with author(s) 
or otheis 


17% 


8% 


10% 


4% 


0% 


10% 


7% 


11% 


0% 


0 % 


8% 


8% 


8% 


Passed document on to 
colleague(s) | 


3o-/ 


25% 


691 


36% 


50% 


49% 


43% 


38% 


35% 


n% 


31% 


45% 


68% 
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.’.E Vri-44. READERS’ EVALUATION OF REVIEWS BY GENERAL USER GROUPS 



QUALITY 



ill" 


k n 


1 us t r . 


Pt . 


Pr i n ■ 


LI fiu. 


! > v ‘ * . 


t Cl 1 1 . 


| Coll. 


Supert . 


Other 


Referent: c 


■ . 'f 


^ t , i r 


Kus . Sp 


Spiv . 


t* i ju I 


Tf if t» . 


if if it . 


Pr-nf . 


A< 1 m i n . 




Admin . 


Mean/ % 


■U| 


I \= 1 9 1 | 


J | 


|N- 161 | 


1 N-20 1 


! -9| 


! .'L-26 1 


( N‘-2 52 1 


[ N=79 ] 


1 N= 3 3 j 


[N-ll; 


[N“597 ] 


♦ 1 


2.42 


2.53 


2.48 


2 . 60 


2 . n 


2.81 


2.54 


2.41 


2.64 


2 . 36 


(2.50) 


•7 


2.79 


2.88 


2.7 5 


2.90 


2.67 


2.81 


2.81 


7.85 


2.91 


2.45 


(2.81) 


iV 

:o 


2.32 


2.28 


2.36 


2.25 


2.22 


2.54 


2.31 


2.27 


2.33 


2.45 


(2.31) 


, w 


2.53 


2.47 


2.59 


2.75 


2.11 


2.69 


2.47 


2.47 


2.58 


2.55 


(2.51) 


• t»7 


2.66 


2.66 


2.70 


2.90 


2.89 


2.77 


2.72 


2.81 


2.88 


2.55 


(2.72) 


.7 


2.31 


2.21 


2.36 


2.40 


2.11 


2.65 


2.32 


2.28 


2.39 


2.18 


(2.32) 




80% 


84% 


79% 


80% 


67% 


85% 


85% 


82% 


85% 


64% 


( 82%) 


. is 


4% 


5% ' 


3% 


5% 


22% 


4% 


3% 


6% 


9% 


9% 


( 4%) 


;rz 


13% 


9% 


13% 


10% 


11% 


8% 


9% 


10% 


3% 


9% 


( 10%) 



UTILITY 





2.67 


2.57 


2.74 


2.75 


2.67 


2.88 


2.79 


2.71 


2.67 1 


2.64 


(2.72) 


" 30 


2.25 


2.10 


2.33 


2.15 


2.22 


2.54 


2.44 


2.30 


2.36 1 


2.18 


(2.33) 


40 


2.61 


2.50 


2.50 


2.65 


2.44 


2.62 


2.62 


2.54 


2.52 


2.55 


(2.58) 


. 50 


2.62 


2.40 


2.62 


2.80 


2.56 


2.88 


2.65 


2.68 


2.76 


2.36 


(2.63) 


23 


2.16 


2.09 


2.26 


2.40 


2.22 


2.27 


2.21 


2.15 


2.12 


2.09 


(2.20) 


03 


2.18 


2.07 


2.17 


2.10 


2.00 


2.15 


2.14 


2.06 


2.00 


2.18 


(2.13) 


.23 


2.42 


2.43 


2.34 


2.35 


2 . 22 


2.50 


2.38 


2.28 


2.27 


2.09 


(2.36) 


. >7 


2.41 


2.33 


2.47 


2.65 


2.33 


2.77 


2.47 


2.49 


2.42 


2.36 


(2.47) 


03 


2.14 


2.19 


2.17 


2.30 


2.33 


2.08 


2.14 


2.09 


2.24 1 


2.09 


(2-14) 



IMPACT 



i % 


18% 


22% 


18% 


407 


0% 


38% 


17% 


22% 


272 


18% 


( 19%) 


, i7 


68% 


36% 


73% 


7 5? 


56% 


81% 


71% 


71% 


762 


55% 


( 69%) 


■ \% 


35% 


33% 


46% 


45? 


0% 


42% 


47% 


48% 


452 


36% 


( 42%) 


0% 


37% 


24% 


36% 


20? 


22% 


50% 


31% 


24% 


15% 


36% 


( 32%) 




10% 


7% 


11% 


0% 


0% 


8% 


8% 


8% 


9% 


182 


( 8%) 




49% 


43% 


38% 


35? 


11% 


31% 


45% 


68% 


552 


36%? 


( 46%) 
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TABLE VII-45. READERS’ OVERALL EVALUATION OF PRACTICAL GUIDANCE PAPERS 



READER EVALUATIONS (N-456) 




QUALITY 


UTILITY 






Mean 




Mean 


Coverage 


2.43 


Relevance 


2.67 


Up-to-dateness 


2.77 


Need 


2.35 


Organization 


2.33 


Comparative usefulness 


2.52 


' . 4 

Writing 


2.53 


Purpose of use: 




Format 


2.74 


Obtain overview 


2.54 


Eiscussion 


2.30 


Look up facts 


2. "24* 






Identify individuals 


2.12 




Percentage 


Identify relevant 


2.26 


Length: 




literature 


About right 


83% 


Update knowledge 


2.41 


Too long 


4% 


Obtain new 


2.18 


Too short 


8% 


knowledge 




IMPACT 








Percentage 




Used to make decision 


23% 




Applied in my work 


65% 




Used to give advice 


49% 




Examined other 


documents 


27% 




Consulted with 


author (s) or others 


6% 




Passed document on to colleague(s) 


50% 
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TABLE VII-46. READERS’ EVALUATION OF PRACTICAL GUIDANCE PAPERS BY LEVELS OF 

EFFORT 







QUALITY 








Low Effort 
[N=86] 


Medium Effort 

[N»167] 


High Effort 

[N-203] 


Reference 

Mean/Percent 

[N-456] 


Coverage 


2.41 


2.38 


2.49 


(2.43) 


Up-to-dateness 


2.77 


2.75 


2.79 


' (2.77) 


Organization 


2.34 


2.32 


2.33 


(2.33) 


Writing 


2.64 


2.53 


2.48 


(2.53) 


Format 


2.83 


2.75 


2.70 


(2.74) 


Discussion 


2.40 


2.24 


2.30 


(2.30) 


Length: 

About right 


81% 


83% 


85% 


(83%) 


Too long 


27 . 


4% 


4% 


( 4%) 


Too short 


10 % 


10% 


6% 


( 8%) 



UTILITY 



Relevance 


2.63 


2.63 


2.72 


(2.67) 


Meed 


2.24 


2.31 


2.43 


(2.35) 


Comparative usefulness 


2.37 


2.48 


2.62 


(2.52) 


Purpose of use: 
Obtain overview 


2.48 


2.44 


2.65 


(2.54) 


Look up facts 


2.16 


2.28 


2.25 


(2.24) 


Identify individuals 


2. 13 


2.14 


2.09 


(2.12) 


Identify relevant 
literature 


2.28 


2.25 


2.27 


(2.26) 


Update knowledge 


2.41 


2.34 


2.48 


(2.41) 


Obtain new knowledge 


2.08 


2.17 


2.24 


(2.18) 



IMPACT 



Used to make decision 


13% 


25% 


25% 


(23%) 


Applied in my work 


57% 


69% 


65% 


(65%) 


Used to give advice 


40% 


46% 


55% 


(49%) 


Examined other documents 


28% 


25% 


29% 


(27%) 


Consulted with author (s) 
or others 


27 . 


8% 


71 


( 6%) 


Passed document on to 
colleague(s) 


48% 


43% 


56% 


(50%) 
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TABLE VII-47 



READERS’ EVALUATION OF PRACTICAL GUIDANCE PAPERS BY LEVELS 
OF VISIBILITY 



QUALITY 




LOW 

Visibility 

[N-77J 


Medium 

Visibility 

TN-1261 


High 

Visibility 

fN-2531 


Reference 

Mean/Percent 

[N-4561 


Coverage 


2.40 


2.35 


2.49 


(2.43) 


Up-to-dateness 


2.79 


2.76 


2.77 


' (2.77) 


Organization 


2.29 


2.33 


2.34 


(2.33) 


Writing 


2.55 


2.52 


2.53 


(2.53) 


Format 


2.74 


2.76 


2.73 


(2.74) 


Discussion 


2.30 


2.25 


2.32 


(2.30) 


Length: 

About right 


92% 


78% 


83% 


(83%) 


Too long 


4% 


3% 


4% 


( 4%) 


Too short 


3% 


13% 


7% 


( 8%) 



UTILITY 



Relevance 


2.71 


2.56 


2.72 


(2.67) 


Need 


2.29 


2.21 


2.44 


(2.35) 


Comparative usefulness 


2.48 


2.40 


2.59 


(2.52) 


Purpose of use: 
Obtain overview 


2.45 


2.43 


2.62 


(2.54) 


Look up facts 


2.21 


2.21 


2.27 


(2.24) 


Identify individuals 


2.08 


2.20 


2.09 


(2.12) 


Identify relevant 
literature 


2.29 


2.24 


2.27 


(2.26) 


Update knowledge 


2.35 


2.33 


2.47 


(2.41) 


Obtain new knowledge 


2.18 


2.15 


2.20 


(2.18) 



IMPACT 



Used to make decision 


27% 


14% 


26% 


(23%) 


Applied in my work 


77% 


57% 


66% 


(65%) 


Used to give advice 


49% 


39% 


53% 


(49%) 


Examined other documents 


21% 


27% 


2P% 


(27%) 


Consulted with author (s) 
or others 


3% 


10% 


6% 


( 6%) 


Fasaed document on to 
colleague(s) 


45% 


43% 


54% 


(50%) 
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TABLE VII-48. READERS’ EVALUATION OF PRACTICAL GUIDANCE PAPERS BY GENERAL 

SUBJECT AREA 



QUALITY 


1 




Instr . 
Content 
[N*144J 


Ed. Adm ./ 
Services 
[N-142] 


Special/ 
Other Educ . 
('•roups 
tN-1141 


Higher 
Educ . 
[N*56] 


Reference 

Mean/Percent 

[N=456] 


Coverage 


2.47 


2.44 


2.41 


2.36 


(2.43) 


Up-to-dateness 


2.71 


2.79 


2.82 


2.79 


(2.77) 


Organization 


2.34 


2.35 


2.32 


2.29 


(2. 33) 


Writing 


2.63 


2.48 


2.54 


2.39 


(2.53) 


Format 


2.77 


2.74 


2.74 


2.68 


(2. 74) 


Discussion 


2.31 


2.30 


2.29 


2.29 


(2.30) 


Length: 

About rif ht 


83% 


86% 


85% 


75% 


(83%) 


Too long 


1% 


4% 


4% 


71 


( 4%) 


Too short 


9% 


6% 


7% 


13% 


(-8%) 



UTILITY 



Relevance 


2.67 


2.70 


2.69 


2. 54 


(2.67) 


Need 


2.30 


2.42 


2.36 


2.29 


(2.35) 


Comparative usefulness 


2.47 


2.52 


2.62 


2.46 


(2.52) 


Purpose of use: 
Obtain overview 


2.49 


2.58 


2.61 


2.45 


(2.54) 


Look up facts 


2.27 


2.26 


2.26 


2.09 


(2.24) 


Identify individuals 


2.10 


2.13 


2.31 


2.11 


(2. 12) 


Identify relevant 
literature 


2.21 


2.28 


2.34 


2.20 


(2.26) 


Update knowledge 


2.42 


2.40 


2.50 


2.27 


(2.41) 


Obtain new knowledge 


2.10 


2.23 


2. 24 


2.18 


(2.18) 



IMPACT 



Used to make decision 


17% 


25% 


27% 


23% 


(23%) 


Applied in my work 


63% 


61% 


73% 


66% 


(65%) 


Used to give advice 


50% 


50% 


47% 


45% 


(49%) 


Examined other documents 


22% 


32% 


31% 


23% 


(27%) 


Consulted with author(s) 
or others 


2% 


9% 


9% 


5% 


( 6%) 


Passed document on to 
colleague(a) 


45% 


50% 


56% 


46% 


(50%) 
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TABLE VI 1-49. READERS' EVALUATION OF PRACTICAL Gl 

USER GROUPS 



QUALITY 





Reading 
Spec . 
IN-10} 


Spue . 
lU\uc . 
[N=9J 


Yoc . 
Lduc . 
[N-fcj 


Su per -- 
v i sor 
IN- 20} 


1 

Coun- 
selor 
IN-25 j 


k n 

Stal 1 
{ N~<»9 ) 


Instr. 
Kos . ?p. 
[N-.UJ 


IT eg . 
Spec . 

. 1 N-83 J 


I'r i n- 
l i pa 1 
LN=11J 


I ' 1 err? . 
IVach . 

iN-n 


S < '( . 

Ttui: h . 
1 N=2U | 


13.1 1 . 
i’rol . 
IN=69J 


Coverage 


2. SO 


^ 2 . U 


2. SO 


2.65 


2 , 24 


7 . 3 S 


2.70 


2.49 


2.27 


3.00 


2.10 


2.4 3 


Up-to-dateness 


2.90 


2.89 


2.83 


2.85 


2. 84 


2.78 


2.85 


2.70 


2 . H 2 


3. 00 


2.80 


2.62 


Organization 


2.50 


2.44 


2.17 


2.40 


2.12 


2.22 


2.39 


2.25 


2.09 


2. 29 


2.35 


2.48 


Writing 


2*70 


2.67 


2.33 


2.70 


2.32 


2.43 


2.61 


2.51 


2.18 


2*43 


2.80 


2.67 


Format 


2.80 


2.89 


2.67 


2.85 1 


2.48 


2.68 


2.67 


2.78 


2.82 


2. 57 


2.85 


2.75 j 


Discussion 


2.30 


2.44 


2.33 


2.50 


2.00 


2.14 


2.48 


2.39 


2.27 


2*14 


2.30 


2.32 i 

i 


Length: 

About right 


90S 


67% 


100% 


7 5% 


72% 


75% 


94% 


89% 


100% 


4 3% 


95% 


| 

80% I 


Too long 


0% 


0% 


0% ' 


5% 


16% 


3% 


3% 


2% 


0% 


29% 


0% 


i% 1 

i 


Too short 


102 


22% 


0% 


10% 


0% 


16% 


3% 


5% 


0% 


02 


5% 


14% ! 

■ 



UTILITY 



Relevance 


2.90 


2.89 


2.33 


2.65 


2.24 


2.64 


2.88 


2.72 


2.55 


2.86 


2.55 


2.75 : 


Need 


2.50 


2.56 


2.17 


2.45 


2.16 


2.29 


2.42 


2.36 


2.36 


2.71 


2.15 


2.41 ‘ 


Comparative usefulness 


2.50 


2. 44 


2.50 


2.55 


2.28 


2.42 


2.64 


2.55 


2.64 


2.57 


2.40 


2.43 


Purpose of use; 
Obtain overview 


2.70 


2.56 


2.50 


2.65 


2.28 


2.54 


2.42 


2.58 


2.55 


2.57 


2.55 


2.46 


Look up facts 


2.50 


2.22 


2.00 


2.15 


2.08 


2.30 


2.33 


2.19 


2.09 


2*57 


2.25 


2.25 


Identify individuals 


2.00 


2.11 


2.00 


2.30 


2.12 


2.10 


2.00 


2.13 


2.00 


2.29 


2.05 


2.10 


Identify relevaut 
literature 


2.10 


2.67 


2.00 


2.35 


2.24 


2.17 


2.21 


2.23 


2.18 


2.43 


2.10 


2.32 


Update knowledge 


2.80 


2.33 


2.33 


2.60 


2.36 


2.32 


2.48 


2.45 


2.64 


2 .57 


2.30 


2.38 


Obtain new knowledge 


2.30 


2.44 


2.00 


2.20 


2.16 


2.14 


2.21 


2.19 


2. 18 


2.29 


2.10 


2.10 



IMPACT 



Used to make decision 


20% 


33% 


33% 


40% 


28% 


16% 


30% 


14% 


45% 


29% 


5% 


16% 


Applied in my work 


90% 


78% 


83% 


65% 


60% 


61% 


67% 


66% 


7 3% 


71% 


45% 


68% 


Used to give advice 


40% 


56% 


50% 


55% 


28% 


42% 


36% 


48% 


91% 


0% 


■ 45% 


57% 


Examined other documents 


20% 


44% 


33% 


20% 


4% 


29% 


12% 


31% 


27% 


0% 


35% 


25% 


Consulted with author(s) 
or others 


0% 


22% 


0% 


0% 


12%. 


12% 


3% 


7% 


0% 


0% 


10% 


62 


Passed document on to 
colleague(s) 


3 OX 


67% 


67% 


40% 


, 64% 


39% 


64% 


52% 


7 32 


14% 


60% 


35% 
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TABLE VI 1-49. READERS' EVALUATION OF PRACTICAL GUIDANCE PAPERS BY GENERAL 

USER GROUPS 



QUALITY 





i mil — 


r; P 


I list r . 


Vn'£. 


i’r i n- 


i: 1 turn . 


Sri . 


Cwi i . 


U.l 1 . 


Suprr l . 


Other 


Reference 


T 


s»* 1 or 


t.u t 


K s . r * | • . 


Spt c , 


rip 1 1 


I'tMC h # 


iV'irh . 


prnt . 


Adm i n . 




Adm i r . 


Mean / : : 


‘M 


|N“J'i] 


t ! 


IN- 31] 


1 N*8 3 J 


LN-liJ 


IN--7] 


( N“20 | 


j (N-69 j 


[N= 19 j 


lN«-29] 


IN-30] 


[N-436] 


; 3 


2 . 24 


2 . 2 5 


2.70 


2.49 


■ » > •? 


.1 . 00 


2 . 30 


2.42 


2 . 4 7 


' r 


> f — 

^ t 


(2.43) 




2 . 84 


2.78 


2.85 


2.70 


2.82 


3.00 


2.80 


2.62 


2.79 


2.93 


2.73 


(2.77) 


,%0 


2.12 


2.22 


2.39 


2.25 


2.09 


2 . 29 


2.35 


2.48 


2.32 


2.38 


2.47 


( 2 . 33 ) 


;o 


2.32 


2.43 


2.61 


2.51 


2.18 


2.4 3 


2.80 


2.67 


2.47 


2.52 

*L . 


2.47 


( 2 . 53 ) 


S 5 


2.48 


2.68 


2.67 


2.78 


2.82 


2.57 


2.85 


2.75 


2.74 


2.76 


2.77 


( 2 . 74 ) 


50 


2.00 


2.14 


2.48 


2.39 


2.27 


2 . 14 


2.30 


2.32 


2.32 


2.31 


2.23 


( 2 . 30 ) 


- >% 


7 22 


75 % 


94 % 


89 % 


100 % 


4 3 % 


95 % 


80 % 


74 % 


90 % 


87 % 


( 83 %) 


5 % 


16% 


3 % 


3 % 


2 % 


0 % 


29 % 


0 % 


i% 


5 % 


7 % 


3 % 


( 4 %) 


[ )% 


0 % 


16 % 


3 % 


5 % 


0 % 


0 % 


5 % 


14 % 


16 % 


3 % 


3 % 


( 8 %) 



UTILITY 



.65 


2.24 


2 . 64 


2.88 


2.72 


2.55 


2.86 


2.55 


2.75 


2.58 


2,69 


2.73 


( 2 . 67 ) 


45 


2.16 


2.29 


2.42 


2.36 


2.36 


2.71 1 


- 2 .15 


2.41 


2.31 


2.34 


2.40 


( 2 . 35 ) 


35 


2.28 


2.42 


2.64 


2.55 


2.64 


2.57 ^ 


2.40 


2.43 


2.79 


2.59 


2.67 


( 2 . 52 ) 


65 


2.28 


2.54 


2.42 


2.58 


2.55 


2.57 


2.55 


2.46 


2.63 


2.52 


2.73 


( 2 . 54 ) 


. 1 5 


2.08 


2.30 


2.33 


2.19 


2 . Q 9 


2.57 


2.25 


2.25 


2.26 


2.21 


2.27 


( 2 . 24 ) 


10 


2.12 


2.10 


2.00 


2.13 


2.00 


2.29 


2.05 


2.10 


2.11 


2.21 


2.13 


( 2 . 12 ) 


55 


2.24 


2.17 


2.21 


2.23 


2 . IS 


2.43 


2.10 


2.32 


2.37 


2.38 


2.37 


( 2 . 26 ) 


60 


2.36 


2.32 


2.48 


2.45 


2.64 


2.57 


2.30 


2.38 


2.42 


2.41 


2.33 


( 2 . 41 ) 


20 


2.16 


2.14 


2.21 


2.19 


2.18 


2.29 


2.10 


2.10 


2.32 


2.24 


2.27 


( 2 . 18 ) 



IMPACT 



0% 


28 % 


16 % 


30 % 


14 % 


45 % 


29 % 


5 % 


16 % 


32 % 


41 % 


30 % 


( 23 %) 




60 % 


61 % 


67 % 


66 % 


73 % 


71 % 


45 % 


68 % 


74 % 


62 % 


57 % 


( 65 %) 




28 % 


42 % 


36 % 


48 % 


91 % 


or 


4 5 % 


57 % 


37 % 


69 % 


50 % 


( 49 %) 




4 % 


29 % 


12 % 


31 % 


27 % 


0 % 


35 % 


25 % 


42 % 


34 % 


27 % 


( 27 %) 


'% 


12 % 


12 % 


3 % 


7 % 


0 % 


0 % 


10 % 


6 % 


5 % 


3 % 


3 % 


( 6 %) 




64 % 


39 % 


64 % 


52 % 


r 

r- 


14 % 


60 % , 


35 % 


53 % 


62 % 


53 % 


( 50 %) „ 
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TABLE VI I- 50. 



READERS ’ 



JB^BRALI 



EVALUATION OF BIBLIOGRAPHIES 



♦ 



/ 



READER EVALUATIONS (N-501) 

QUALITY 



Coverage 

Up-to-dateness 

Organization 

Format 

Textual material 



Mean 

2.49 

2.78 

2.23 

2.72 

2.47 



No. of references: 
About right 
Too many 
Too few 



Percentage 

81% 

4% 

11 % 



Mean 
2.77 
2.39 
2.70 

Purpose of use: 

To identify documents on particular topics 
To identify documents on particular projects 
To identify documents by particular individuals 
To identify documents from particular institutions 
To perform comprehensive search of literature 
To see kinds of new work being reported 



\ 

' UTILITY 

*• " ' “'....A- 

Relevance 

Need 

Comparative usefulness 



Percentage 

73% 

41% 

13% 

11 _% 

55_% 

67% 



IMPACT 

Were cited documents examined? Yes 381 (76%) Was content of cited 

document (s) as expected 

from bibliographic 

reference? Yes 73 % No 27 % 




V 

VII-109 

4 v 



* 



t 



TABLE VI I- 51 



READERS' EVALUATION OF BIBLIOGRAPHIES BY LEVEL OF EFFORT 





QUALITY 








Low Effort 

[N-237] 


Medium Effort 
[N*112 ] 


High Effort 
[N-152J 


Reference 

Mean/Percent 

[N-501] 


Coverage 


2.48 


2.47 


2 . 30 


(2.49) 


Up-to-dateness 


2.80 


2.70 


2.81 


(2.78) 


Organization 


2.19 


2.29 


2.23 


(2.23) 


Format 


2.70 


2.71 


2.7b 


(2.72) 


Textual material 


2.46 


2.47 


7.^7 


(2.47) 


No. of references; 










About right 


82% 


81% 


78% 


(817 ) 


Too many 


4/. 


3% 


3% 


( 42) 


Too few 


10% 


11% 


14% 


(11%) 



UTILITY 



Relevance 


2.77 


2. 78 


2. 75 


— — ■ 

(2.77) 


Need 


2.37 


2.47 


2. 35 


(2.39) 


Comparative usefulness 


2.72 


2.63 


2.73 


(2.70) 


Purpose of use: 

To identify documents 
on particular topics 


73% 


7'Z 


72 X 


(73%) 


To identify documents 
on particular projects 


41% 


46% 


35% 


(41%) 


To identify documents 
by particular individuals 


15% 


9% 


14% 


(13%) 


To identify documents 
from particular institutions 


11% 


1 3% 


8 ? 


( 11 %) 


To perform comprehensive 
search of literature 


55% 


59% 


52% 


(55%) 


To see kinds of new 
work being reported 


65% 


70% 


66 7 * 


(67%) 



IMPACT 



Were cited documents 










examined? (Yes) 


75% 


IhZ 


80% 


( 767.) 


Was content of cited 










document's) as expected 


Y*=72% 


Y=7i% 


Y=76% 


(Y= 7 3%) 


from bibliographic 
reference? (Yes/No) 


N=28% 


N=29% 


N=24% 


(N-27%) 
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TABLE VI 1-5 2 



READERS’ EVALUATION OF BIBLIOGRAPHIES BY LEVEL OF VISIBILITY 



QUALITY 





Low 

Visibility 

fN-2441 


Medium 

Visibility 

FN-2301 


Trip 

Visibility 

tN«27L_ 


Reference 

Mea ^-!5i1 nt 


Coverage 


2.50 


2.48 


2.44 


(2.49) 


Up-to-dateness 


2.80 


2.74 


2.89 


(2.78) 


Organisation 


2.23 


2.24 


2.07 


(2.23) 


Format 


2.73 


2.73 


2.59 


(2.72) 


Textual material 


2.51 


2.43 


2.41 


(2.47) 


No. of references: 
About right 


84% 


78% 


81% 


(81%) 


Too many 


3% 


5% 


4% 


( 4%) 


Too few 


9% 


U% 


7% 


(11%) 



UTILITY 



Relevance 


2.75 


2.77 


2.85 


(2.77) 


Seed 


2.39 


2.39 : 


2.33 


(2.39) 


Comparative usefulness 


2.73 


2.69 


2.59 


(2.70) 


Purpose of use: 

Tc identify documents 
on particular topics 


73% 


72% 


70% 


(73%) 


To identify documents 
on particular projects 


43% 


38% 


41% 


(41%) 


To identify documents 
by particular individuals 


13% 


14% 


11% 


(13%) 


To identify documents 
from particular institutions 


l':% 


10% 


7% 


(11%) 


To perform comprehensive 
search of 1 iterature 


54% 


59% 


37% 


(55%) 


To see kinds cf new 
J work being reported 


68% 


63% 


81% 

i 


(67%) 

■ 



IMPACT 



Here cited documents 
examined? (Yes) 


72% 


81% 


70% 


(76%) 


Has content of cited 
document(s) as expected 
from bibliographic 
reference? (Yea/No) 


Y«69% 

N-31% 


Y«/7% 

N-23% 


Y-67% 

N-33% 


(Y-73%) 

(N-27%) 
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TABLE VI I- 53 



READERS’ EVALUATION OF BIBLIOGRAPHIES BY GENERAL SUBJECT AREA 



QUALITY 




Instr . 
Content 
TN-1741 . 


Ed . Adm * / 

Services 

fN-1611 


Special/ 
Other Educ 
Groups 
[N=145] 


Higher 
Educ . 
[N*21] 


Reference 

Mean/Percent 

[N-501] 


Coverage 


2.50 


2.44 


2.54 


2.38 


(2.49) 


Up-to-dateness 


2.76 


2.80 


2.76 


2.90 


(2.78) 


Organization 


2.19 


2.23 


2.28 


2.24 


(2.23) 


Format 


2.73 


2.71 


2.72 


2.81 


(2.72) 


Textual material 


2.41 


2.48 


2.53 


2.38 


(2.47) 


No. of references: 
About right 


80% 


78% 


82% 


100% 


(81%) 


Too many 


5% 


4% 


3% 


0% 


( 4%) 


Too few 


11% 


14% 


9% 


0% , 


(11%) 



UTILITY 



Relevance 


2.75 


2.79 


2.77 


2.76 


(2.77) 


Need 


2.37 


2.34 


2.45 


2.43 


(2.39) 


Comparative usefulness 


2.71 


2.66 


2.72 


2. 76 


(2.70) 


Purpose of use: 

To Identify documents 
on particular topics 


73% 


73% 


73% 


67% 


(73%) 


To Identify documents 
on particular projects 


41% 


36% 


44% 


43% 


(41%) 


To Identify documents 
by particular Individuals 


19% 


9% 


12% 


10% 


(13%) 


To identify documents 
from particular Institutions 


7% 


12% 


13% 


14% 


(11%) 


To perform comprehensive 
search of literature 


63% 


53% 


50% 


38% 


(55%) 


To see kinds of new 
work being reported 


67% 


67% 


66% 


71% 


(67%) 



IMPACT 



Were cited documents 
examined? (Yes) 


79% 


80% 


68% 


76% 


(76%) 


Was content of cited 
document(s) as expected 
from bibliographic 
reference? (Yea/No) 


Y-75% 

N-25% 


Y-76% 

N«24% 


Y-66% 

N-34% 


Y-76% 

N»24% 


(Y-73%) 
(N=27%) . 
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r ^ 

C, . J 



tabu: v i i - 5 . 



READERS’ EVALUA' 






QUALITY 





K tMd 1 ng 

Spi ,- . 

[ N- 10 j 


Spvc . 
Kduv . 
IN=1«1 


rduc . 
IN- 5| 


hupt’f- j 
visor | 
[ N= 1 3 j I 


Lmm- 
sv lor 

|n=hj 


T 

i; /, ii 

St.it t 
i V‘o3| 


I ns t r . 
Res, Sp. 
[ N“83 1 


1 o-K. 
Spiv . 

i:*'-9uj 


: t l tl 

‘ ipnl 
[N lui 


Coverage 


3. 30 


2.56 


2.40 


2.4 7 ! 

, - 4 


2.36 


2.47 


3.41 


2.47 


3. Ml 


Up-to-dateness 


2.60 


2.72 


2.60 


2.67 J 


2.27 


2.72 


2.78 


2.82 


3.00 


Organization 


2.00 


2.22 


2.00 


2.33 


1.91 


2.26 


2.29 


2.18 


2.30 


Format 


2.90 


2. 78 


2.60 


2.67 


2.36 


2.70 


2.69 


2.69 


2.90 


Textual material 


2.10 


2.56 


2.20 


2.80 


2.45 


2.36 


2.52 


2.38 


2.70 


No. of references: 
About right 


100% 


83% 


80% 


87% 


91% 


85% 


72% 


84% 


90% 


Too many 


0% 


6% 


0% 


0% 


0% 


2% 


7% 


0% 


ox 


Too few 


0% 


11% 


0% 


7% 


0% 


11% 


18% 


10% 


10 % 



UTILITY 



Relevance 


3.00 


2.78 


2.80 


2.80 


2.45 


2.68 


2.80 


2.77 


3.00 


Need 


2.60 


2.28 


2.80 


2.47 


2.09 


2.47 


2.27 


2.38 


2 60 


Comparative usefulness 


2.60 


2.94 


2.40 


2.73 


2.55 


2.66 


2.56 


2.70 


2.70 


Purpose of use: 

To identify documents 
on particular topics 


40% 


83% 


80% 


80% 


100% 


70% 


74% 


79% 


401 


To identify documents 
on particular projects 


30% 


44% 


60% 


60% 


64% 


47% 


26% 


48% 


50% 


To identify documents 
by particular individuals 


20% 


17% 


20% 


33% 


9% 


9% 


6% 


19% 


30% 


To identify documents 
from particular institutions 


0% 


28% 


20% 


20% 


0% 


13% 


8% 


16% 


30% 


To perform comprehensive 
search of literature 


70% 


56% 


100% 


53% 


73% 


64% 


48% 


52% 


901 


To see kinds of new 
work beiu,} reported 


60% 


612 


100% 


53% 


100% 


60% 


62% 

1 


72% 


40% 



IHPACT 



Were cited documents 
examined? (Yes) 


90% 


72% 


100% 


80% 


55% 


77% 


62% 


83% 


80% 


Was content of cited 
docuaent(s) as expected 


Y*60% 


Y=61% 


Y=100% 


Y=80% 


Y=55% 


Y=75% 


Y-62% 


Y=79% 


Y 70% 


from bibliographic 
reference? (Yes/No) 


N=40% 


N=39% 


N- 0% 


N=20% 


N=4 5% 


N=25% 


N=38% 


N-21% 


N-30% 



REAPERS' EVALUATION OF BIBLIOGRAPHIES BY GENERAL USER GROUPS 



QUALITY 








— T 

■» 1 ; 

it ! 

J 

' • / 


i lisp- . 
Res. Sp. 
j V s', ! 


!' 1 • 
Spi . 

iN” v bj] 


Pt i ii- 
o i p.i 1 
[ N - I t; 1 


i'. 1 fin . 
Teach • 

1 N- 10 ! 


Si*i . 

Teach . 
IN-27] 


Cel 1 . 
Prof . 

[ N-* 108] 


Coll . 
Acim i n . 
[NO 3] 


Super t . 
IN-13] 


ot her 

Admin. 

[‘>16] 


Ref i-rt nc o 
Mean/ 7 
IN^SOI] 


.* . *« i 


2 . *4 7 


2 . 6 U 


2.80 


2.52 


2 . 58 


2.31 


2.50 


2.20 


(2.49) 


7 ^ 


.’.78 


2.82 


3.00 


2.90 


2.70 


2.85 


2.85 


2.94 


2.47 


(2.78) 


. :e> 


P . p u 


2.18 


2.30 


2.20 


2.30 


2.29 


2.00 


2.31 


1.87 


(2.23) 


.70 


2.69 


2.69 


2.90 


2.90 


2.63 


2.81 


2.69 


2.75 


2.60 


(2.72) 


.3b 


2.52 


2.38 


2.70 


2.50 


2.30 


2.56 


2.54 


2.63 


2.27 


(2.47) 


857 


72% 


84% 


90% 


50% 


89% 


81% 


69% 


88% 


53% ' 


.%) 


21 


7% 


0% 


0% 


10% 


7% 


5% 


0% 


0% 


27% 


( 4%) 


117 


187 


10% 


10% 


10% 


0% 


13% 


23% 


13% 


7% 


( 11%) 



UTILITY 



.68 


2.80 


2.77 


3.00 


2.70 


2.78 


2 . b 1 ' 


2.62 


2.94 


2.33 


(2.77) 


: .47 


2.27 


2.38 


2.60 


2.30 


2.37 


2.43 


2.23 


2.50 


2.40 


(2.39) 


’ .66 


2.56 


2.70 


2.70 


2.60 


2.81 


2.81 


2.77 


2.75 


2.40 


(2.70) 


70% 


74% 


79% 


40% 


40% 


74% 


76% 


85% 


81% 


33% 


( 73%) 


47% 


26% 


487 


50% 


20% 


37% 


44% 


31% 


38% 


13% 


( 41%) 


9% 


6% 


19% 


;o% 


10% 


26% 


12% 


0% 


13% 


IX 


( 13%) 


i3% 


8% 


16% 


30% 


10% 


7% 


7% 


8% 


6% 


IX 


( 11%) 


64% 


48% 


52% 


90% 


30% 


37% 


60% 


54% 


50% 


47% 


( 55%) 


60% 


62% 


72% 


40% 


50% 


85% 


68% 


77% 


63% 


53% 


( 67%) 



IMPACT 



77% 


62% 


83% 


80% 


60% 


78% 


85% 


77% 


88% 


40% 


( 76%) 


' - 75% 


Y-62% ; 


Y=797 


Y 70% 


Y=60% j 


jY— 78% 


Y*81% 


Y=77% 


Y=88% 


Y*33% 


(Y-73%) 


n - 25% 


N=38% 


N=21 7 


N=30% 


N=40% 


N=22% 


N-19% 


N=23% 


N=13% 


K*67% 


(N*27%) 
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TABLE VII-55. SPECIALISTS v OVERALL EVALUATIONS OF REVIEWS 



SPECIALISTS' EVALUATIONS (N* 208) 



QUALITY 





Excellent 


Good 


Fair 


Poor 


Not 

Applicable 


No 

Response 


Choice of author 


40% 


39% 


7% 


1% 


6% 


6% 


Selection of 

content/material 


43* 


39% 


12% 


4% 


0% 


2% 


Choice of references 


44% 


40% 




2% 


1% 


?% 


Inclusion of current 
material 


46% 


36% 


9% 


4% 


1% 


3% 


Accuracy 


46% 


45% 


3% 


2% 


3% 


1% 


Interpretation 


33% 


37% 


17% 


9% 


2% 


2% 


Organization 


42% 


41% 


11% 


4% 


0% 


1% 


Organization of 
references 


40% 


42% 


8% 


3% 


0% 


1% 


Format 


21% 


47% 


22% 


3% 


5% 


2% 


Writing 


40% 


45% 


14% 


1% 


0% 


1% 



UTILITY 



Would you recommend to colleagues? Usefulness for Various Purposes 



Yes 92% No 8% 



If yes: 

Purpose of Use (N“191) 


Very 

Useful 


Somewhat 

Useful 


Not At 
All Useful 


No 

Response 


Obtain overview 


80% 


20% 


1% 


0% 


Look up facts 


31% 


56% 


12% 


2% 


Identify relevant literature 


— m — 




3% 


1% 


Identify individuals or institutions 


51% 


41% 


8% 


1% 


Update knowledge 


45% 


47% 


8% 


1% 


Obtain new knowledge 


51% 


36% 


11% 


2% 


Obtain practical guidance 


44% 


45% 


9% 


2% 


Other : 


6% 


1% 


0% 


93% 











Need for Document of This Type 




Overall Usefulness of Document 


59% Very great 




64% it is a very useful document. 


38% Moderately great 




29% it is not unusually useful, but 


3% Not at all great 




it is worth having available. 






7% Its usefulness is too limited 






to justify its publication. 
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TABLE Vil-56. SPECIALISTS’ OVERALL EVALUATIONS OF PRACTICAL GUIDANCE 

PAPERS 

SPECIALISTS' EVALUATIONS (N“94> 



QUALITY 





Excellent 


Good 


Fair 


Poor 


1 No t 

npp] icable 


No 

Response 


Choice of author 


38% 


32% 


7% 


5% 


12% 


5% 


Selection of 

content /material 


40% 


30% 


17% 


6% 


6% 


0% 


Choice of references 


30% 


35% 


12% 


7% 


15% 


1% 


Inclusion of current 
material 


34% 


35% 


18^ 


9% 


3% 


1% 


Accuracy 


36% 


36% 


10% 


4% 


12% 


2% 


Interpretation 


34% 


36% 


10% 


9% 


10% 


27 , 


Organization 


35% 


43% 


10% 


9% 


2% 


2% 


Organization of 
references 


23% 


40% 


6% 


4% 


19% 


6 % 


Format 


26% 


J8% 


2b% 


9% 


0% ' 


2% 


Wri ting 


43% 


36% 


12% 


4^ 


4% 


1% 



UTILITY 



Would you recommend to colleagues? 
Yes 33% No T T4 



Usefulness for Various Purpuses 



If yes: 

Purpose of Use (N-‘78) 


r 

Very 

Useful 


Somewhat 

Useful 


Not At 
All Useful 


No 

Response 


Obtain overview 


72% 


24% 


4% 


0% 


Look up facts 


36% 


47% 


17% 


0% 


Identify relevant literature 


46% 


32% 


17% 


5% 


Identify individuals or institutions 


42% 


32% 


21% 


5% 


Update knowledge 


44% 


45% 


9% 


3% 


Obtain new knowledge 


50% 


39% 


12% 


0% 


Obtain practical guidance 


69% 


26% 


5% 


0% 


Other : 


8% 


4% 


0% 


89% 











Need for Document of This Type 

48% Very great 
^8% Moderately great 
3% Not at all great 



Overall Usefulness of Document 

53% It Is a very useful document. 
30% It Is not unusual!;; useful, but 



it is worth having available. 

] 77 

Its usefulness is too limited 
to justify its publication. 
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TABLE VI I- 5 7 



X 



SPECIALISTS’ OVERALL EVALUATIONS OF BIBLIOGRAPHIES 
SPECIALISTS' EVALUATIONS (N-12Q) 



QUALITY 





Excellent 


Good 


Fair 


Poor 


Not 

Appl icable 


No 

Response 


Choice of author 


26% 


32% 


9% 


5% 


21% 


a 


Selection of 

content /material 


23% 


33% 


15% 


54 


22% 


27 , 


Choice of references 


26% 


34% 


17% 


3% 


14% 


5% 


Inclusion of current 
material 


33% 


40% 


.12% 


12% 


4% 


oz 


Accuracy 


39% 


33% 


5% 


1% w 


21% 


21 


Interpretation 


16% 


34% 


6% 


4% 


39% 


27 , 


Organization 


23% 


40% 


5% 


9% 


23% 


n 


Organization of 
references 


31% 


39% 


9% 


6% 


y% 


17 . 


Format 


19% 


50% 


19% 


9% 


3% 


1% 


Writing 


29% 


45% 


8% 


2% 


16% 


1% 

U -d 



UTILITY 



Would 5 iu recommend to colleagues? 
Yes 88% No 



Usefulness for Various Purposes 



If yes: 

Purpose of Use (N«114) 


Very 

Useful 


Somewha t 
Useful 


Not At 
All Useful 


No 

Response 


Obtain overview 


61% 


27% 


7% 


4% 


Look up facts 


29% 


47% 


20% 


4% 


Identify relevant literature 


71% 


28% 


0% 


1% 


Identify individuals or institutions 


46% 


49% 


5% 


0% 


Update knowledge 


44% 


45% 


9% 


3% 


Obtain- new knowledge 


30% 


48% 


17% 


5% 


Obtain practical guidance 


38% 


40% 


19% 


3% 


Other : 


5% 


1% 


0% 


94% 




- 







Need for Document of This Type 

513* Very great 

Moderately great 
53* Not at all great 



Overall Usefulness of Document 
50% It is a very useful document. 

^3% it is not unusually useful, but 
it is worth having available. 

Its usefulness is too limited 
to justify its publication. 



6 % 
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TABLE VI I -58 



A. TABLE OF SAMPLE SIZES 
OVERALL EVALUATIONS 




Reviews 

957 


Practical 

Guidance 

Papers 

456 


Bibliographies 

501 




Low 


349 


86 


237 


By Level of Effort 


) Med. 


406 


167 


112 


> 


High 


202 


203 


152 


> 


| Low 


342 


77 


244 


By Level of Visibility 


> Med. 


314 


126 


220 


> 


1 High 


301 


253 


27 






301 


144 


174 


1 


l 2 


200 


142 


161 


By Subject 


f 3 


174 


114 


145 


J 




282 


56 


21 














1 


6 


10 


10 




2 


12 


9 


18 




3 


29 


6 


5 




4 


25 


20 


15 




5 


30 


25 


11 




6 


191 


69 


53 


By User Group 


, 7 


58 


S3 


85 




8 


151 


83 


90 




9 


20 


11 


10 




10 


9 


7 


10 




11 


26 


20 


27 




12 


252 


69 


108 




13 


79 


19 


13 




14 


33 


29 


16 


J 


15 


11 


30 


15 



B. REFE 




To Use : 

1. Fi 

2. Ta 

3. Fi 
di 
le 




C 



J 



TABLE VI 1-58. 



REFERENCE table for computing significance tests 



B. REFERENCE TABLE FOR APPROXIMATING SIGNIFICANCE DIFFERENCES 



Bibliographies 

501 

237 

112 

152 

244 
230 
27 ' 



174 

161 

145 

21 



10 

IS 

5 

15 
11 
53 
85 
90 
10 
10 
27 

108 

13 

16 
15 



Required Differences in Means 
for Approximate Sample Sizes 


STATISTICAL 

SIGNIFICANCE 

LEVEL 


10 


20 


30 


75 


150 


225 


300 


500 

or 


.62 


.47 


.33 


.24 


.17 


.14 


.12 


.09 


.01 


.46 


.37 


.26 


.20 


.13 


.12 


.09 


.06 


.05 



To Use: 



1 

2 , 

3. 



Find the difference between the two means being compared; 

Take the average of the two sample sizes involved (Table A) ; 

Find the closest approximate sample size in Table B and check the 
difference that is required in that column for either the .01 or .05 
level of confidence. 
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VIII. SUMMARY FADINGS v CONCLUSIONS, AND RECOMMENDATIONS 



This study has clearly demonstrated that NCEC information analysis products 
are — in varying degrees — known and read and, on the whole, are favorably 
received by the survey-respondent populations. 

A second and equally significant conclusion is that there is still a great deal 
of room for improvement in alerting the educational community to information 
analysis products and in facilitating its access to these products. 

These two conclusions and others of importance are discussed more fully in the 
following three sections, which focus on the major study issues, subsidiary 
study issues, and implications for future survey methodology. In these dis- 
cussions, reference is made to sections in previous chapters that contain data 
supporting the stated conclusions, or the relevant data are su mm arized in the 
text. Recommendations are made only to the extent that the survey findings 
point toward the need for specific followup policy decisions or actions. 

In examining our findings, the reader should keep in mind that no discipline 
has well-recognized standards for the adequacy of information dissemination.* 
This means that there is no generally agreed-upon framework for judging the 
adequacy of information exchange among professional communities or between 
information-dissemination units, such as NCEC, and the potential users of the 
information.** For example, it is not possible to say, a priori, whether 



it 

There are some data available in the literature on diffusion and adoption of 
innovations, but we do not believe that they can be applied directly to this 
study. 

itit 

For planning purposes, the project staff formulated some expectations along 
these lines. For example, we anticipated that one out of four respondents to 
the Screening Questionnaire (Ql) would be a product user and, therefore, a 
potential product evaluator. This was a gross underestimation of .'he actual 
ratio of users to non-users, which was almost 7 to 1. 
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10, 50, or 90 percent of the population studied should have been able to report 
familiarity with NCEC products. Of those who were familiar, should 10, 50, or 
90 percent have actually read one or more of the 10 products they were 
asked to evaluate? Should 10, 50, or 90 percent of the expressions of user 
satisfaction with the products be above the midpoint on the evaluation 
scales? 

The present study provides useful baseline data for positing goals for future 
information analysis activities and for establishing standards of excellence 
for future products. 

A. MAJOR STUDY ISSUES 

Several questions to be addressed in this study were identified in the 
Introduction to this report. Summaries of the general findings relative 
to these questions are presented belox*. The discussion reflects only the 
findings for this study r s respondent populations, but we believe that the 
survey methodology as a whole, the sampling procedures, rates of returns, and 
ultimate sample sizes provide a strong base for drawing these general conclusions 
and recommendations. 

1. ARE USERS FAMILIAR WITH NCEC INFORMATION ANALYSIS PRODUCTS? 



REFERENCE: Chapter V, Sections A and B 

The question of respondents’ familiarity with NCEC products is addressed at both 
general and product-specific levels. 

General Familiarity . Among the total 3,013 respondents to the Screening Question- 
naire, 2,526 or 87 percent, were familiar with (i.e., had heard about or had read) 
products from at least one NCEC unit. Of the non-random sample only, 94 percent 
reported prior familiarity with products; surprisingly, of the random sample, 

72 percent reported prior familiarity. This finding does not support the conclusions 
drawn in the recently completed Indiana University study,* which found that there 



* 

Bernard M. Fry, Final Report: Evaluation Study of ERIC Products and Services , 

Indiana Univ' rsity, March 1972. 
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is very little rvareness of NCEC information analysis products among ERIC users. 
Assuming that the populations were reasonably similar, one possible explanation 
for the different results is that the Indiana study respondents were not sure 
which products are specifically called "interpretive summaries" or "research 
reviews," These terms, as "information analysis products" itself, are peculiar 
to information science and can best be understood by example. The color insert 
in Q1 of examples of products and the document representations ifi. Q2 — which are 
important features of the SDC survey methodology — may have provided our survey 
participants with the necessary stimuli for a more solid and accurate basis for 
recalling the materials. This interpretation is supported by the fact that some 
Q1 respondents indicated that they were ERIC users but did not know the partic- 
ular products shown in the color insert. 

Some specific findings suggest that the high level of general familiarity found in 
the present study may be somewhat inflated. For example, a considerable number 
of Q1 respondents (21 percent for the random sample and 41 percent for the non- 
random sample) indicated familiarity with EMC bibliographies and were classed 
as NCEC "users." Yet, on a product-specific level, their apparent level of 
familiarity with EMC bibliographies was not borne out. Although these biblio- 
graphies are widely disseminated directly by the USOE and indirectly through 
GPO, the universe of EMC bibliographies is sufficiently small that (to validate 
the rates found in Ql) the rate of recognition by evaluators (in Q2) should 
have been higher. It is possible that the term "Educational Materials Center 
Bibliographies" is so general (there are numerous Educational Materials Centers 
throughout the country) that some respondents may have assumed a familiarity 
that did not in fact exist. 

There is little to be gained in speculating on the exact number of false 
recognitions reflected in the level of familiarity data. The main finding — 
that there is a high level of general familiarity with NCEC products— is 
substantially borne out by the ensuing "hit rate," i.e., matches of specific 
products with actual readers, in the Evaluation Questionnaire Survey. 
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Product-Specific Familiarity . Over half (approximately 60 percent) of the 
1,251 respondents contributed at least one Reader evaluation to the total of 
1,914 Reader evaluations ^across all product types). This percentage representation 
was comprised equally of random and non-random sample participants. It is quite 
probable that this hit rate might have been even better, since the 146 products 
selected by the project staff for the study represented only a part of the whole 
range of NCEC products. How much better, we do not know. What is known, however, 
is that the data by cases, i.e., by each individual and his package of 10 documents, 
strongly demonstrate the potential for even greater use of NCEC products — in 
69 percent of the cases (across all product types), products had neither been 
heard of nor read. The potential for greater use is supported by the generally 
high : .iv els of need and potential relevance reported by Non-readers in apprais- 
ing NCEC products brought to their attention in the study. 

The need for improving awareness of NCEC products was shown even more dramatically 
in the document-by-document analysis. For example, a Practical Guidance Paper 
on adult basic education and English as a second language, published in 1969, 
received strongly favorable reviews from all three Specialists.* Although it is 
apparently a very significant contribution to the literature, it had been read 
or skimmed by only 2 percent of the 55 users who were asked to review it; 84 
percent had neither heard about it or read it. 

The responses of the Non-user population provide another indication of the 
potential for increased use. Although this population might have been expected 
to exhibit some general apathy toward information in their responses to the 
need and reasons for not reading 11 items, they did not. In approximately 
two-thirds of the cases, the documents brought to their attention appeared to 
ue potentially useful, even very useful, and in no cases did Non— users who 
had previously heard about a product indicate a lack of interest in it; their 
chief problem was that they could not readily obtain a copy. 



* 

Patricia Hefferman— Cabrera * A Handbook for Teachers of English t o Non— English 
Speaking Adults , October, 1969. (Document No. 14) 



VIII-4 



Both Non-reader and Non-user data indicate a twofold problem: 

1. There is a general lack of awareness of NCEC products on 
a product-specific level. 

2. There is a general belief that the products are not 
easily accessible. 

This problem causes an under-utilization of products that are demonstrably of 
potential value to educators. We recommend that action be initiated to develop 
a three- to five-year plan for improving both the alerting function and the 
delivery system associated with NCEC products. These plans should include 
specific targets reflecting reasonable increments of utilization and product 
recognition found in the present survey. 

The need for improved alerting procedures was specifically noted by several 
respondents in their general comments. The alerting function needs to include 
an announcement system that goes beyond the current procedures, such as PREP 
Briefs (disseminated through state education agencies), listings in ERIC 
Products or Research in Education , or newsletter announcements. Some of the 
document representation and document assi* ment techniques used in this project 
may be useful as a basis for developing a Selective Dissemination of Informa- 
tion (SDI) approach.* 

The solution to the delivery problem is not so clear-cut, because the reasons 
for not using NCEC products may, in some instances, have as much to do with 
local factors as with product delivery mechanisms. For example, there were 
comments from respondents that indicated budget problems (the "cost of hard 
copy is prohibitive”) or a reluctance to use microfiche. It is possible that 



★ 

Two o l' pr USOE-supported projects could provide additional valuable information 
in planning improved alerting procedures. Under a separate contract, SDC is 
carrying out research aimed at defining appropriate partitions of the ERIC 
file. This study involves an analysis of educator information needs and ERIC 
document characteristics. The Stanford University Institute for Co mmun ication 
Research is carrying out the second study of the problems of developing a 
sensing network to help maintain continuing awareness of user information 
needs . 
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several of the solutions (including simplification of the EDRS order procedures, 
increased dissemination through professional association agencies and journals, 
and extension of support from USOE to local and state educational information 
centers for a more direct delivery system) will need to be considered. 

Plans for evaluating these strategies should be incorporated into the overall 
’’plan for growth" in the information analysis products program. An evaluation 
scheme is discussed briefly in Section C of this chapter. 

2. ARE PRODUCTS MEETING NEEDS OF EDUCATORS? 



REFERENCE: Chapter VII, Section C 



Survey respondents were asked to express the intensity of their needs for 
particular documents. On a scale in which 2.00 represented "moderately great; 
the topic is of continuing importance to me" and 3.00 represented "very great; 
had an immediate need for a document on this topic," the mean responses were 
2.36 for Readers and 2.49 for Specialists. While the reasons for the somewhat 
different responses from the two groups are not clear, it does seem that the 
NCEC products reviewed were meeting' some of the needs, particularly the con- 
tinuing information needs, of the educators who used them. 

There are several ways to interpret these results with regard to Readers* 
intensity of need. For example, it seems possible that the more urgent needs 
are those associated with the most current issues and perhaps the ' issues were 
less represented among topics covered in the • roduct sample. It is also 
possible that the notion of "immediate need" jL not an apt one to describe an 
educator’s typical information requirements. In either case, the results seem 
positive, particularly in view of the fact that information analysis products 
necessarily rely upon the prior establishment of a literature base or other 
evidence of trends in educational research and practice. 
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Another dimension of "need" was investigated through the questionnaire item on 
"uses” and "degree of usefulness" associated with NCEC products. Specialists 
indicate that the products would be useful for several purposes, including 
obtaining new knowledge. On the other hand, the data from Readers indicate 
that they use these products primarily to obtain overviews and to update knowl- 
on subjects already known to them. Obtaining new knowledge was not claimed to 
be a major use. The use and usefulness data from this survey should be of con- 
siderable help to NCEC in targeting documents for particular intended uses, as 
well as for, specific audiences. 

NCEC products seem to be meeting the needs of the educators who use them. This 
positive finding, in conjunction with the "comparative usefulness" data, indi- 
the value and continued importance of the information analysis program. 

No finding supports this conclusion more dramatically than the high (2.7 on a 
3-point scale) ratings of comparative usefulness for Bibliographies. 

There is no way to infer from the study data anything about needs that are not 
being met by products, either in terras of subject coverage or product types. 
Some of the co mmen ts allude to a "continuing need" for a product in a given 
area or to a need for products from one clearinghouse that are comparable to 
those from another, but rarely was the need for topics not covered by the 
products specifically expressed by participants. Therefore, there is no basis 
for recommending that topic or product-type coverage be altered in any way. 

If there is one major recommendation for action that can be drawn from these 
findings, it is that careful study be given to each document evaluation by the 
appropriate NCEC originating units. There is much useful information on the 
documents (and the topic coverage) from the respondents, particularly concern- 
ing those products that should be updated regularly. This kind of information 
should be of considerable use to the NCEC units in planning the allocation of 
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resources to areas of continuing need and for identifying the time-frame for 
periodic updating. If a certain fraction of the resources available to each 
NCEC unit were earmarked for products associated with continuing needs, the 
remaining resources could be devoted to enlarging this coverage and adding 
more current topics. The development of such a plan for "periodic coverage" 
may also help the announcement function, for a user who receives one issue 
of a series will be on the alert for forthcoming issues. 

3. WHAT KINDS OF IMPACT, IF ANY, ARE THE PRODUCTS HAVING? 

%r 

The major impact of the products is in the application of information to the 
users’ work; they are presumably contributing to educators' professional 
growth and development. But, beyond this general and rather expected impact, 
it would appear that NCEC products are having two other kinds of impact: 

1) a specific problem-solving impact and 2) an informational impact. 

The survey data indicate that the NCEC products help educators in a variety of 
ways to solve specific problems. This variety can best be seen in the comments 
of Readers that are reported in detail in Volume II of this report. 



‘ " " - ' — — — * 

REFERENCE: Chapter VII, Secti .r D 



There are two major evidences of the informational impact: 

• In approximately half of the cases, users are major channels 
for further dissemination of Reviews and Practical Guidance 
Papers. These data reinforce the Indiana University study 
(cited on page VII-2) finding that a most important way in 
which individuals become aware of ERIC is through their 
colleagues. 

o The NCEC products themselves are reported to be promoting 
further use of information and perhaps more discriminating 
reading patterns. Both the Reviews and Bibliographies help 
to move readers from the source product to other documents. 
In the case of Bibliographies, the survey data show that 
this function is being served well and that the contents of 
the examined documents are generally as could be expected 
from the contents of the bibliographic citations. 
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There is no clear evidence that information in products is used specifically in 
decision-making situations. Although percentages reported for this use were 
relatively low, the possible overlap of this category with other impact areas 
makes it difficult to draw any definite conclusions. 

4. HOW WELL ARE THE PRODUCTS RECEIVED BY THEIR USERS AND INTENDED AUDIENCES? 

REFERENCE: Chapter VII, Sections B, F, and I 



The overall evaluations of documents, across all product types, clearly indicate 
that the products are favorably received and, from the comments, are appreciated 
by the respondent population. In general, the overall ratings for each quality- 
related item show that: 

• For users' needs, documents cover topic (s) better than 
"moderately well;" 

• The discussions are somewhat better than "reasonably thoughtful;" 

• The organization (or classification of entries in Bibliographies) 
is somewhat better than "satisfactory;" 

• The writing in Reviews and Practical Guidance is better than 
"moderately clear;" 

• The formats are generally "helpful to readability (or usability) 
and understanding;" 

• The textual materials (e.g., summaries, annotations) in Biblio- 
graphies are better than "moderately useful;" 

• The length of Reviews and Practical Guidance Papers and number 
of references in Bibliographies are "about right;" 

• The materials are generally believed to have been up-to-date 
(at publication) . 

The analyses by user group do not clearly demonstrate that any particular user 
groups are less satisfied than others. These data do, however, indicate some 
relative differences in judgments on certain quality-related areas among user 
groups. Although a small sample, Elementary Teachers are often on the extreme 

of the overall means, e.g., this group rated the clarity of writing for- Reviews 
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relatively low. On the other hand, Principals rate the clarity of writing 
in Reviews higher than they do that in Practical Guidance Papers, lhese general 
indicators of differences among user groups and differences recorded for 
specific documents (see Chapter VII, Section F) are useful in examining the 
targeting practices for particular products and product types. 

No general patterns of deficiencies are evident in the special analyses by general 
subject area, but again, there are indicators of relatively lower ratings in 
specific subject areas for certain quality dimensions. This analysis was probably 
weakened by the fact that the groupings — unfortunately — had to be broadly devised 
to ensure a sufficient number of evaluations. 

One conclusion that might be drawn from these findings is that while the overall 
evaluations are positive, more can still be done to improve the specific qualities 
of particular product types. Without detracting from the overall evaluations, we 
believe that there is invaluable information contained in the individual document- 
evaluation profiles (contained in Volume II) that can be most helpful to the NCEC 
units in the preparation of future documents. Some general concerns are expressed 
repeatedly for all of the product types. There are also user comments made on 
specific products that point out more specifically the high or low quality elements. 

0 

We recommend that NCEC (and the staffs of each unit) study the individual 
document evaluation profiles with the documents in hand (preferably an original 
and an EDRS hard copy)* to identify exemplary and deficient products and the 
factors contributing to their quality. From these assessments, it will be 
possible to develop a quality-control checklist for each step in the creation 
cycle, from conception through publication. Such quality -assurance checklists 
could have a major impact on ERIC products by forcing continuous attention to 
the functional goals of each product’s format and content. Decisions to accept, 
reject, or revise a particular product, or to interrelate it more thoroughly 



Several references were made by Specialists to the poor quality of reproduction. 
Since some of them used SDC-produced photocopies of documents, each criticism 
was checked and verified as stemming from a hard copy received from EDRS. It is 
possible that some original copies were not adequate for microfilming, which is 
a serious consideration in the production cycle of future products. 
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with others, would be sharply illuminated by a formal means for asking and 
recording answers to several types of questions. 



For example, What purpose is to be served by this product ? strictly alerting, 
satisfying demand search, exhaustive search, critical comparative review, recent 
state-of-the-art informative review, applications adoption information, etc.? 
What should be the treatment criteria : Descriptive naming only, description 

without comparison, detailed evaluative comparisons, how— to— do-it materials 
and sources? What kind of author or authors are needed : "National perspective" 

individual, representatives of typical applied situations, "idea men" and 
generators of innovative techniques, system-analytic and cost-effectiveness 
thinkers, etc.? What should be the calendar and bread th-of -coverage scopes : 

Very narrow and very recent, broad and recent, narrow and extended, etc.? 

There could be legitimate exceptions in the adherence to checklists encompassing 
these elements, because of limitations of funds or uniqueness of the material 
or purpose, but the decision to ignore an item can then be made ft with purpose" 
and this purpose can then be explained to the reader in the foreword of a 
product. 

A major thread that runs through the comments for all product types is a divert. 
sity of expectations that cannot possibly be met by one or even se veral products 
on a given topic . Therefore, a clear statement of purpose, scope, audience, 
and limitations is required in the products and, perhaps, as a part of the 
descriptor system, so that the potential use of the product can be identified 
by the user and so that the product will not be as vulnerable to the multipli- 
city of individual expectations. Examples of the elements of choice, drawn from 
the variety of comments for each product type, further illustrate these points. 

• For Reviews . The variety of ways in which Reviews are used 
and the different groups who use them, suggests the need to 
develop a family of Reviews for specific purposes, i.e., 
for overviews vs. in-depth treatments, or for updating 
knowledge vs. obtaining new knowledge. 
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Some criticisms x^ere levelled at the analysis and inter- 
pretation sections of Reviews. Reviews involving expert 
analysis and critical interpretations require larger 
investments in authors* fees and time. Perhaps the number 
of reviews with critical analyses and interpretations should 
be limited and supplanted by well-organized and annotated 
bibliographies — particularly where limitations in resources 
permit only a "quasi-review.** Some useful insights into 
the one or several resolutions to this problem might be 
found in a further study of the evaluations of the Junior 
College Review Series and the Educational Management Clearinghouse’s 
Analysis and Bibliography Series , both of which are well-represented 
in the product sample. 

The extent to which Reviews are used by practitioners, and not 
solely by researchers, suggests that Reviex^s also need to be 
thought of in terms of different audiences. Different levels 
of treatment may be required, and the level should be explained, 
along xtfith the rationale for scope and selection of material, 
in the foreword of each product. 

Practical Guidance Papers . There were enough differences in 
reactions to the formats of PREP reports that those which are 
particularly attractive to readers could be singled out and 
perhaps used as models for future products. Their levels of 
treatment on topics also suggest one possible model for 
practitioner-oriented products in general. 

Some of the Practical Guidance Papers, e.g., those from the 
Counseling and Personnel Services Clearinghouse, are formatted 
in an unusual style, which on appearance seems useful fur 
quick and easy reading. These formats received mixed reviews, 
and it would be interesting to compare comments with the products, 
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and build upon what appears to be a unique approach. Some 
exemplary products should be looked at, particularly those about 
which the Specialists were unanimous in their comments. Two 
products of this kind are the already mentioned H andbook for 
Teachers of English to Non-English Speaking Adults (Document 14) 
and Preparing to Teach Economics: Sources and Approaches 

(Document 112) . 

• For Bibliographies . Bibliographies are particularly vulnerable 
to differences in expectations of readers. Criticism was made 
of omissions, incompleteness, and the inclusion (or exclusion) 
of classic entries. A clear statement of intended coverage 
is needed, along with the criteria for selection, the time- 
period covered, and the intended function (i.e., current awareness 
vs. a comprehensive search). Particular mention was made of 
the usefulness of annotations. The intermediate function that 
the annotated bibliography can serve, as opposed to an uncritical 
brief review, makes it a fairly attractive product- '.ype. It can 
be done by a professional bibliographer, and claims need not be 
made for intensive analysis and interpretation, an element 
requiring the involvement of subject specialists. 

Several kinds of comments were directed toward the organization 
of bibliographies. Not all classification schemes or organizations 
of entries can be accommodated, and. some further study of those 
products in which differences were tv; pressed should be examined 
more closely to see if there are any patterns of preferences. 
Although special-request bibliographies are usually scanned by 
the user to locate strongly desired specific types of items, 
the standard periodic type of bibliography is perhaps 
more usually scanned by the user for items of more casual 
"maintenance" interest to him. With such motivation, 
and with many other pressing demands on his time, a user's 
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tolerance for scanning many items to find the few of 
direct Interest to him may not be very high. Therefore, care- 
ful sorting and formatting of periodic bibliographies by de- 
ti led topi-ial breakdown can do much to improve the alerting 
impact of periodic bibliographies. 
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ANALYSES BY LEVELS OF EFFORT AND VISIBILITY 



ERIC! 



Two special analyses were concerned with measurements of cost (level of effort) 
and distribution (level of visibility). One major conclusion can be drawn 
from these analyses: that there is no conclusive evidence, using this study’s 

measurements of effort and visibility, that either element is strongly correlated 
with the quality, utility, or impact of products. 

There is, however, much in the findings of this study to suggest that the 
quality and utility of NCEC products result from factors other than those 
considered in these two indexes. Some of these factors are perhaps more 
closely related to the products’ conception and the care taken in the early 
(and last) stages of their development. Study of the "product development 
environment" was outside the scope of this project, but this study’s findings 
and recommendations strongly support the necessity for pursuing questions of 
cost-effectiveness . 

Level of Effort . We recommend that NCEC and its various product-originating 
units work together with the individual document evaluation profiles to 
begin the process of identifying key elements in the entire production cycle 
that contribute to the creation of exemplary products. (This exercise is an 
extension of the one recommended in the preceding section.) By tracing through 
the development of several major products in this study (and perhaps some later 
ones that are known to have been well-received more recently) , the basis for 
developing a refined cost-related index can be established. The results of 
these efforts should help to establish the framework for asking for each product: 
How many of the methods, arrangements, and decisions associated with exemplary 
products are relevant to improving its cost-effectiveness. 

Level of Visiblity . The concept of visibility should be considered further 
from two different perspectives. First, there is some indication (see Section 
H of VII) that visibility may be related to quality. If we assume that it is 
planned visibility (or exposure) that affects quality (i.e., knowledge that a 
product will have high visibility may place pressures for ^-xcellence on the 
entire production effort), then a plan for increasing the use of products 
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may, in itself, contribute an added psychological investment to the level of 
effort. 

Second, for the development of an improved announcement and delivery system, 
the question still remains as to which of the current announcement and dis- 
tribution methods, individually or in combination, are effective in reaching 
intended users. This question was directly addressed in the Indiana University 
study. Those findings can be supplemented, again, by an individual document 
review of Volume II and matching announcement/distribution procedures with 
recognition data. For example, bibliographies from one clearinghouse were not 
known by this study’s respondent populations although they appeared in ERIC 
columns of professional journals and had the highest visibility index for this 
study. On the other hand, an annually compiled bibliography that appears in one 
user group’s major professional periodical was well-known. As indicated earlier, 
we do not believe that the delivery system problem is easily solved, for any 
solution must accommodate apparent differences among user groups in professional 
traditions, information user patterns, and involvements with professional 
associations. 



c. 



RECOMMENDATIONS FOR A CONTINUING METHODOLOGY FOR EVALUATING NCEC 



INFORMATION ANALYSIS PRODUCTS 

This study was actually a composite of several substudies. The two major 
objectives were: 1) to assess the outreach of products and 2) to 

evaluate their quality and utility. In the future, we recommend that the 
two evaluations be conducted separately so that the design study for one 
does not limit nor unnecessarily expand on the design of the other. 

The following discussion of design considerations assumes the best of all 
possible worlds; l.e., that the quality and usefulness of the NCEC information 
analysis products are recognized, that the program's continued existence is 
ensured, and that increase* in NCEC product utilization is planned, adequately 
funded, and implemented. 

1. ASSESSMENT OF THE OUTREACH OF PRODUCTS 

A periodic assessment, similar to the one in this study, should be made of ♦‘he 
outreach of products, particularly within a year or so of the implementation 
of an announcement system or of modifications to the delivery system. The 
plan for the design -of such an assessment should be incorporated into the overall 
plan for growth and for continuing to understand the users and potential user 
community. Several design components, drawn largely from the survey methodology 
described in Chapter III, are suggested for consideration in the following 
discussion. 

Survey Participants . Although there was evidence that either a random or 
non-random sample could provide a broad cross-section of educators, we believe 
that there is merit in continuing the dual sampling approach. It might be 
useful, for example, to select as primary sampling units states that are not 
served by local or state educational information centers and to retain 
distinctions among all sampling units (e.g., state educational information 
centers, state agencies, and school districts). Any significant differences 
in awareness or use could then be detected, and resources could be allocated 
to remedy the problem. 
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The random and non-random samples also provide a framework for futher study of 
educators as information users. Although we were able to combine these two 
groups for purposes of analyzing their evaluation data, the earlier (Q1 ) analyses 
that were conducted separately on their interests and familiarity with NCEC 
units (see Chapters IV and V) raise some questions of interest, particularly in 
regard to dissemination strategies. For example, the non-random sample (drawn 
from mailing and user lists of agencies in the "NCEC family") shows greater 
breadth of interest than the random sample. The reasons for this difference 
merit further exploration: 

• Is greater contact with information in itself a "need creating" 
force that leads users to peripheral or related interest areas? 

• Do more frequent users of information simply learn to state 
their needs more broadly (not just in terms of their role/ 
function or setting) in order to identify required information? 

Insufficient background information was obtained from participants in this study 
to pursue these questions. However, additional items could be added to the 
Screening Questionnaire that would provide for their further consideration. 

Survey Instrument . Some changes are recommended for the present Screening 
Questionnaire; such changes might include the following: 

• Background variables . There will continue to be a problem in 
accommodating different terminologies in job positions, role 
perceptions, and dual roles and functions among educators. 
However, it would be useful for product developers (e.g. , the 

clearinghouse staffs) to develop a typology to be used in a 

* 

questionnaire that would adequately reflect their needs in 
thinking about intended audiences. For example, a more direct 
match of roles with subject area interests may be required 
for the content-area clearinghouses, so that they can 
distinguish between program specialists in the science area 
and secondary teachers in the social studies area. 
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Other items might be added to the questionnaire, such as 
frequency of use and typical information sources used* 

• The Screening Question . Additional stimuli should be added 
to the present item 4, in Ql, that reflect any changes in 
announcement procedures (e.g., flyers or brochures), and 
that thoroughly cover the channels through which respondents 
might have obtained copies of the products. (There was some 
evidence that respondents did not have a clear picture of the 
ERIC system, f-jr at the bottom of the list of clearinghouses, 
they would add; "What about (RISE), (BAIC) , (Florida ERIC), 
(Boulder Center)?" These are, of course, either state or 
local educational information centers that are disseminators, 
rather than originators, of ERIC materials. 

We strongly encourage the continued use of color reproductions 
of sample products as a method of prompting accurate recall. 

The questionnaire should be brief, for we believe that this characteristic was 
a major factor in the satisfactory return rate. 

2. CONTINUING EVALUATION OF PRODUCTS 

On the basis of what we consider to have been two successful survey approaches, 
we recommend a two-fold approach to the continuing evaluation of products: 

1) a post-publication review system and 2) a periodic evaluation survey. 

Post-publication Review System . The willingness of Specialists to contribute 
to this study and the thought and care taken by most of them in responding to 
the questionnaire suggest that this resource in the educational community 
should be tapped further. One way in which t^iey might be used is in a po°*"- 
publication review of selected products. As pointed out in Chapter V, the 
number of products that had been reviewed in the professional literature was 
small. Thus, the post-publication review would not only serve to continue the 
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in-depth evaluations of the products' quality, but it would extend the announce- 
ment service of the publications through the professional journals and, pre- 
sumably, enhance the follow-up dissemination. 

Specialists in this study were selected from individuals recommended by the 
ERIC clearinghouse advisory boards. These advisors and appropriate professional 
associations should be a continuing source for recommendations of individuals 
who are representative of the targeted audiences of products. It might also be 
useful with certain products to have several reviews from specialists repre- 
senting different groups (e.g., researchers and practitioners). 

Evaluation Survey . We believe that this study's Evaluation Questionnaire Survey 
serves as a sound guide for developing a continuing methodology for an NCEC-wide 
evaluation of products by general users. However, a few changes are recommended 

• The evaluation should be tied directly to known product 
users, identified through State and local educational 
information centers, clearinghouse lists, and EDRS. 

(Through an appropriate coding system, each unit could 
note product users throughout the year and, thereby, 
facilitate the entire identification process. For 
example, EDRS might use a special code for individual 
purchasers.) The diversity that we found in the ERIC 
clearinghouse mailing lists suggests that their useful- 
ness for purpose of identifying product users must be 
qualified on an individual-clearinghouse basis. For 
example, some lists that we used are professional 
membership or special subscription lists that clearly 
identify individuals who are regular recipients 
of certain products. In other cases, mailing lists are 
used basically for announcement purposes (e.g., for 
newsletter distribution) and do not necessarily represent 
product users; in these cases, users must be drawn from 
request files, rather than from mailing lists. 
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* The survey instrument should be refined, as necessary, to 
include the most important elements of quality and utility, 
as identified through the document—by— document studies 
recommended earlier. It would also be useful to study fur- 
ther the relation between quality and utility, for as indicated 
in Chapter VII, there is some evidence that use may be a fairly 
independent variable and that the need for a document on a 
particular topic may supersede the need for a quality document. 

• A second survey instrument should be developed for the 

information intermediaries, so that they not only evaluate 
documents relevant to their own professional interests, 
but that they are also provided with more meaningful items 
that explore the utility of products for their particular 
service role. 

Together, these studies should help to ensure that the products remain sensitive 

to the present needs of users and that the shifts in needs and expectations are 
detected. 
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APPENDIX A 

LISTING OF DOCUMENTS IN PRODUCT SAMPLE 



Bibliographic citations for each product in the sample include the following 
information: 



• Type * Product Types: R ■ Review 

P ■ Practical Guidance Paper 
B ■ Bibliography 

• Effort . Level-of-Effort Index: H » High 

M » Medium 
L » Low 

• Visibility , Level-of-Visibility Index: H * High 

M ■ Medium 
L * Low 



• Subject , General Subject Area: 

— Instructional Content 
— Educational Administration 
and Services 

— Special and Other Educational 
Groups 

— Higher Education 



Asterisks after effort or visibility information indicate that no data, or 
only partial data were available; mean hours or copies for that product type 
were assigned. 



1. PREP REPORTS 



1* Treating Reading Difficulties: The Role of the Principal, Teacher , 
Specialist, Administrator Preps 2, 3, 4, 5 , Carl B. Smith, et al. , 

1970. (single edition from GPO: OE-30026; separate monographs; 

ED034 078, EDQ34 079, ED034 080, ED034 081) 

Type = P Effort = H Visibility = H Subject = Instr. Content 

1A. Treating Reading Difficulties: Reading and the Home Environment . 

The Principals Responsibility . 

IB* Treating Reading Difficulties: Establishing Central Reading 

Clinics. The Administrator’s Role . 

lc * Treating Reading Difficulties: Correcting Reading Problems in 

the Classroom. 

1D * Treating Reading Difficulties: Trea ting Reading Disabilities. 

The Specialises Role . 

2. Bilingual Education , Prep 6, Horacio Ulibarri, et al. , 1969. 

(ED 034 082) 

Type = P Effort = H Visibility = H Subject = Spec. /Other Groups 

3* Job-Oriented Education Progra ms for the Disadvantaged. Schools and 

Industry Cooperate. Prep 9 , Trudy W. Banta, et al. , 1969. (ED 034 085) 

Type = P Effort = H Visibility = H Subject = Spec. /Other Groups 

4 * Paraprofessional Aides in Education. Prep 12. Carl H. Rittenhouse. 

1969. (ED 034 906) 

Type = P Effort => H Visibility = H Subject = Ed. Adm. /Services 

5 * Sharing Educational Services. Prep 13 . Ray Jongeward and Fran Heesacker, 
1969. (ED 036 666) 

Type = P Effort = H Visibility = H Subject = Spec. /Other Groups 

6 * Social studies a^c he Disadvantaged. Prep 1 4. Jonathon C. McLendon, 
et al., 1970. ( "ft J37 588) ~~ 

Type 83 P Effort - H Visibility = H Subject = Instr. Content 



7. Individualized Instruction. Prep. 16 Jach V. Edling, 1970. (ED 041 185) 
Type * P Effort = H Visibility - H Subject - Ed. Adm. /Services 



2. EMC BIBLIOGRAPHIES 

8. Books Related to English Language and Literature in Elementary and 

Secondary Schools, Lois B. Watt, Delia Goetz, and Caroline Stanley (Comp.), 
October 1969. (GPO: 0E-30024; ED 039 236) 

Type * B Effort - H Visibility * M Subject - Instr. Content 

9. Science and Mathematics Books for Elementary and Secondary Schools , Lois 

B. Watt, Delia Goetz, and Eunice von Ende (Cofcy?.), February 1970. (GPO: 
0E-29071-ED 041 760) \ 

Type « B Effort - H Visibility * M Subject'^/Instr. Content 

10. Books Related to Adult Basic Educ&tion and Teaching English to Speakers of 

Other Languages , Myra H. Thomas, Thelma M. Knuths, Sidney E. Murphy (Comp.), 
May 1970. (GPO: 0E 13039; ED 043 850) 

Type * B Effort - M Visibility - M Subject - Spec. /Other Groups 

11. Education Literature of t he Profession , Eunice von Ende (Comp.), July 1970. 

(GPO: 0E-10060-A; ED 046 890) 

Type - B Effort * H Visibility - M Subject - Ed. Adm. /Services 



3. EltIC CLEARINGHOUSES ON . . . 

ADULT EDUCATION 

12. Resident - al Adult Education: Current Information Sources, No. 25 , 

October 1969, (ED 032 449) 

Type - B Effort - L Visibility « L Subject - Spec. /Other Groups 

13. Physical Facilities in the Education and Training of Adults , Roger DeCrow, 
March 1970. (ED 036 677) 

Effort * L Visibility * L Subject * Spec. /Other Groups 
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Type * R 



14 . 



A Handbook for Teachers of English to Non-English S peaking Adults, 
Patricia Hefferman -Cabrera, October 1969. (ED 033 335) 



Type * P -Effort = M Visibility = M Subject = Spec. /Other Groups 

15. Community Service and Continuing Education: A Literature Review 

James B. Whipple * July 1970. (ED 038 550) 

Type = R Effort = H Visibility = M Subject = Higher Ed. 

16. Educati on for Aging: A Review of Recent Literature , H. Lee Jacobs 

et al. , July 19/0. fED Ubb 5b2) “ ~ 

Type = R Effort = H Visibility = L Subject = Spec. /Other Groups 

17 . Needs — Of People and Their Communities — And the Adult E ducator, 

Ernest E. McMahon, July 1970. (ED 038 551) 

lype as r Effort = M Visibility = L Subject = Spec. /Other Groups 

18. Parent, Home and Family Life Education: Current I n formation Sources N _ o _ . — 30^ 

July 1970 . (ED 039 376) 

Type » B Effort = L Visibility = L Subject = Instr. Content 

19. The Preparation of Adult Educators: A Selected Review of t he Literature 

in North America , Coolie Verner, et al. , September 1970. (ED 041 180) 

Type - R Effort = L Visibility = L Subject = Higher Ed. 

20. Mass Media in Public Affairs Adult Education: A Review of the Literature, 

Hilton M. Power, November 1970. (ED 042 075) 

Type = R Effort = L Visibility = L Subject = Spec. /Other Groups 

COUNSELING AND PERSONNEL SERVICES 

21. Orientation Approaches to Increase Student Awareness of Occupational 
Options , Nancy Sloan, November 1969. (ED 033 255) 

Type = B Effort = L Visibility = L Subject = Instr. Content 

22. A Set of Generalizations and Implications. . . : Guidance a nd Student 

Services for the Culturally Different , G. R. Walz, D. K. Harrison, 

March 1970. (ED 037 596) 

Type = P Effort = L Visibility = M Subject = Sp^r. /Other Groups 



23 . rarpp.r Guidance Practices In Sch ool and Community., Lorraine S. Hansen, 
et al., 1970. (ED 037 595) 



Type 



R 



Effort « & Visibility - M Subject » Ed. Adm. /Services 



24. wp-c.ruitlng the Hard-To-Employ , Personnel Services 2 > 

Perspectives on Training the Disadvantaged: The Hard-To Empl oys 

Don K. Harrison, May 1970. (ED 038 560) 



Type 



Effort = M Visibility « L Subject = Spec. /Other Groups 



2S The Hard-T o-Employ--Who are they ? Personnel Services Review, Series 2, 
Perspectives on Training the Disaivantaged: The Hard-To-Emp . l oz, 

k. Harrison & D. R. Brown, May 1970. (ED 038 559) 



Type 



Effort = L Visibility * L Subject - Spec. /Other Groups 




Type = P Effort = M Visibility = L Subject - Higher Ed. 

27 ^novations in the Training a n d Supervision of Coups. elors ; 1 31 cro-C 0 unsell n£, 

Personnel Services Review, Series 1, Juliet V. Miller, March 1970. 

(ED 030 672) 

Type . p Effort = M Visibility * L Subject - Higher Ed. 

disadvantaged 

28. frt r.-IRCD Resources on the School Dropp^. ERIC-IRCD Urban Disadvantaged 

Series, #14, Adelaide Jablonsky, April 1970. (ED 037 589) 



Type 



B 



Effort » M Visibility “ L Subject - Spec. /Other Groups 



29. The .Tob Corps! A Review of t he ERIC Literature, ERIC-IRCD Urban Disad- 
vantaged Series, Humber 13, Ade laide Jablonsky, March 1970. (ED 036 662) 



Type = B 



Effort = M Visibility - L Subject - Spec. /Other Groups 



30 The w ^.hherhnod Youth Corps : A R eview of the ERIC Literature, ERIC-IRCD 

30 ' urban Disadvantaged Series, Humber 1 2, Adelaide Jablonsky, 5a7ch 1970. 

(ED 036 661) 



Type = B 



Effort * M 



Visibility = L Subject * S^ec. /Other Groups 



A- 5 







3 



31. School Dropout Programs: A Review of the ERIC Literature , ERIC-IRCD Urban 

Disadvantaged Series, Number 10, Adelaide Jablonsky. (ED 035 779) 

Type = B Effort = M Visibility = L Subject = Spec. /Other Groups 

32. The School Drop-out: A Review of ERIC Literature , ERIC-IRCD Urban 

Disadvantaged Series, Number 9, Adelaide Jablonsky, March 1970. 

(ED 035 778) 

Type = B Effort = M Visibility = L Subject = Spec. /Other Groups 

33. The School Dropout and the World of Work: A Review of the ERIC Literature , 

ERIC-IRCD Urban Disadvantaged Series, Number 11, Adelaide Jablonsky. 

(ED 035 780) 

Type * B Effort = M Visibility = L Subject = Spec. /Other Groups 

34. Immigrants and the Schools: A Review of Research . ERIC-IRCD Urban 

Disadvantaged Series, Number 8, David K. Cohen, December 1969. 

(ED 033 263) 

Type = R Effort = L Visibility = H Subject « Spec. /Other Groups 

35. Education. Ethnicity, Genetics, and Intelligenc e. IRCD 
Numter 4, Edmund W. Gordon (Ed.), Fall 1969. (ED 037 519) 

Type “ R Effort « L Visibility = H Subject = Spec. /Other Groups 

36. Media for Teaching Afro-American Studies , IRCD Bulletin, Vol. 6, Nos. 1 and 2 
Adelaide Jablonsky, Spring/Summer 1970. 

Type = R Effort = H Visibility = I T Subject = Instr. Content 

37. Significant Trends in Education of the Disadvantaged , ERIC-IRCD Urban 
Disadvantaged Series, Number 17, Edmund W. Gordon, August 1970. 

(ED 040 305) 

Type = R Effort = L Visibility = H Subject = Spec. /Other Groups 

38. Mutability of Intelligence and Epidemiology of Mild Mental Retardation , 
ERIC-IRCD Urban Disadvantaged Series, Number 18, Zena Stein and Mervyn 
Susser, September 1970. (Reprint) 

Effort = L Visibility = H Subject = Spec. /Other Groups 



Type = R 



EARLY CRILDHOOr EDUCATION 



39 . Multi-Ethnic Books for Head Start Children: Black & Integrated Literature , 

Doris White, July 1969* (ED 031 312) 

Type = B Effort = M* Visibility = M Subject = Spec. /Other Groups 

40. Books in Preschool; A Guide to Selecting, Purchasing, and Using Children 1 s 

Books , Louise Griffin, 1970. (ED 038 178) 

Type ® P Effort * M* Visibility ■ H Subject - Spec. /Other Groups 

41. Sentimentality in Preschool Teachers i Some Possible Interpretations , 

Lilian G. Katz, March 1970. (ED 035 792) 

Type * R Effort = L Visibility - M Subject - Spec. /Other Groups 



EDUCATIONAL MANAGEMENT 

42. ERIC Abstracts Series Number One; Collective Negotiations in Education , 
August 1969. (ED 035 978) 

Type = B Effort - L Visibility - M Subject - Ed. Adm. /Services 

43 . Annotated Bibliography on SchooL-Community Relations , Kathleen 0. Jackson, 
July 1969. (ED 030 220) 

Type * B Effort ® H Visibility * L Subject ® Ed. Adm. /Services 

44 . ERIC Abstracts No. 9: Educational Assessment , September 1970. (ED 044 770) 

Type - B Effort - L Visibility « M Subject » Ed. Adm. /Services 

45 . Directory of Organizations and Personnel in Educational Administration , 

2nd Edition, Stuart C. Smith (Comp.), Sept. 1969. (ED 033 467) 

Type = P Effort » M Visibility - M Subject * Ed. Adm. /Services 

46. Optimum School District Size , Research Analysis Series, #1, Michael E. 

Rickey, December 1969. (ED 035 108) 

Type = R Effort * R Visibility - L Subject,® Ed. Adm. /Services 

47. Program Budgeting and the School Administrator? A Review of Dissertations 
and Annotated Bibliography , Philip K. Piele and David G. Bunting, Sept. 1969. 
(ED 035 065) 



Visibility ® L 



Subject ® Ed. Adm. /Services 
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Type * R 



Effort = R 



48 . 



Status and Scope of Collective Bargaining in Public Education , 

M. Chester Nolte, Sept. 1970. (ED 043 100) 

Type = R Effort - H Visibility = L Subject = Ed. Adm. /Services 

49. Educational and Social Demands on the Schools , Analysis and Bibliography 
Series #1, September 1970. (ED 043 110) 

Type = R Effort = L Visibility = L Subject = Ed. Adm. /Services 

50. Alternative Organizational Forms , Analysis and Bibliography Scries #2, 
September 1970. (ED 143 111) 

Type = R Effort = L Visibility = L Subject ~ Ed. Adm. /Services 

51. Models for Rational Decision Making , Analysis and Bibiiography Series #6, 

John S. Hall, September 1970. (ED 043 115) 

Type = R Effort = L Visibility = L Subject = Ed. Adm. /Services 

52. Linking Schools and State Education Departments to Research and Development 
Agencies, Analysis and Bibliography Series #9, September, 1970. 

(fiD 043 118) 

Type = R Effort = L Visibility = L Subject = Ed. Adm. /Services 

53. The Humanities in Preparing Educational Administrators , Robin H. Farquhar, 
December 1970. (ED 044 765) 

Type = R Effort = H Visibility * L Subject = Higher Ed. 

54. Legal Aspects of Control of Student Activities by Public School Authorities 
(Officials ) , #1 in the NOLPE Monograph Series, E. Edmund Reutter, Jr., 
December 1970. (ED 044 829) 

Type = R Effort = H Visibility = M Subject = Ed. Adm. /Services 

55. Social and Technological Change: Implications for Education , Philip K. Piele 

(Ed.) et al., December 1970. (ED 044 833) 

Type = R Effort = H Visibility - M Subject = Ed. Adm. /Services 

55A. Nature of Our Changing Society: Implications for Schools. 



55B. 



Teacher Militancy: Implications for the Schools . 



55C. System Approaches to Education: Discussion and Attempted Integration * 

55D. System Approaches to Educational Planning . 

55E . Educational Management Information Systems: Progress and Prospectives . 

EDUCATIONAL MEDIA AND TECHNOLOGY 

56. 1 Instructional Materials Center , Don n. Coombs, et al., Dec. 1969. 

(ED 034 438) 

& 

Type - B Effort * L Visibility - L Subject - Ed. Adra. /Services 

57. Museums and Media: A Basic Reference Shelf, & Museums and Media? A Status 

Report , Philip C. Ritterbush & Richard Grove, Dec. 1970. (ED 044 935) 

Type = B Effort - H Visibility * L * Subject - Ed. Adm. /Services 

58. The Interview: An Educationa l Research Tool, Andrew Collins. December 

1970. (ED 044 931) 

Type * P Effort ■ L Visibility m L Subject * Ed. Adm. /Services 

59. A Position Paper on CAI Research and Development . John H. Feldhusen 
& Paul Lorton, Jr., February 1970. (ED 036 204) 

Type * R Effort =■ L Visibility » L Subject - Instr. Content 

60. A Guide to the Literature on Interactive U&e Of Computers for Instruction , 
Karl L. Zinn & Susan McClintock, January 1970. (ED 036 202) 

Type * R Effort - M Visibility « M Subject * Instr. Content 

EXCEPTIONAL CHILDREN 

61* Grouping for Instruction, Exceptional Children Bibliography Series . 

November 1969. (ED 036 034) 

Type «* B Effort * L Visibility - L Subject - Spec. /Other Groups 



62 . Programs for the Mentally Retarded, Exceptional Children Bibliography 
Series , September 1969. (ED 036 029) 



Type * S Effort * L Visibility * L Subject = Spec. /Other Groups 

63. Teaching Exceptional Children, Vol, 2, No. 3 , Spring 1970. 

Type = P Effort = M* Visibility = M* Subject = Spec. /Other Groups 



HIGHER EDUCATION 

64. Urban Universities and the City, Review 2 , David E. Sumner, April 1970. 
(ED 038 556) 

Type = R Effort * H Visibility = L Subject = Higher Ed. 

65. Compendium Series of Current Research, Programs, and Proposals , 

Number 2; Preparing College Teachers , Carol Shulman, August 1970. 

(ED 041 179) 

Type ° R Effort = H Visibility L Subject = Higher Ed. 

66. The Crisis of Purpose: Definition and Uses of Institutional Goals , 

Richard E. Peterson, October 1970. (ED 042 934) 

Type * R Effort =* M* Visibility = M Subject = Higher Ed. 

67. Preventing College Dropouts: A Review , James Harvey, November 1970. 

(ED 043 799) 

Type = R Effort - M Visibility = M Subject = Higher Ed. 

68. Consortia in American Higher Education. Report 7 , Lewis D. Patterson, 

November 1970. (ED 043 800) 

Type “ R Effort * M* Visibility = M Subject = Higher Ed. 

69. Due Process in the Student-Institutional Relationship , Thomas C. 
Fischer, July 1970. (ED 041 189) 

Type = P Effort = M* Visibility = H Subject = Higher Ed. 

70. College Compensatory Programs for Disadvantaged Students, Report 3 , 
William T. Trent, September 1970. (ED 042 932) 

Type = P Effort * M Visibility = M Subject = Higher Ed. 
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JUNIOR COLLEGES 



71. A Developmental Research Plan for Junior College Remedial Education; 

Number 3: Concept Formation . John R. Boggs, August 1969. (ED 032 072) 

« 

Type - P Effort - M Visibility « M Subject « Higher Ed. 

72. Identifying the Effective Instructor , Edward F. O'Connor, Jr. and 
Thomas Justiz, January 1970. (ED 035 416) 

Type * P Effort » M Visibility * L Subject - Higher Ed. 

73. Junior College Research Review; Faculty Recruitme nt, Vol. 4, No. 1 
Dale Gaddy, September 1969. (ED 032 864) 

Type « R Effort - L Visibility - H Subject « Higher Ed. 

74. Junior College Research Review: College-Community Relations , Vol. 4, 

No. 3, Barton R. Herrscher and Thomas M. Hatfield, Nov. 1969. 

(ED 032 888) 

Type - R Effort - M Visibility - H Subject - Higher Ed. 

75. Junior College Research Review; Paying for Junior College s, Vol. 4, 

No. 8, Erick L. Lindman, April 1970. (ED 038 124) 

Type * R Effort - L ^Visibility * H Subject ■ Higher Ed. 

76. Junior College Research Review: Curriculum , Vol. 4, No. 6, February 1970, 

Type ■ R Effort » H Visibility m H Subject * Higher Ed. 

77. Junior College Research Review: Co-operative Work-Experience Education 

Programs in Junior Colleges , Vol. 5, No. 2, Marcia A. Boyer, October 1970. 

(ED 042 455) 

Type - R Effort » L Visibility - H Subject - Higher Ed. 

78. The Junior College Research Review: Occupationally Oriented Students . 

Vol. 5, No. 3, K. Patricia Cross, Nov. 1970. (ED 043 328) 



Type « R 



Effort » M Visibility * H Subject ■ Higher Ed. 



LANGUAGES AND LINGUISTICS 



79 • ACTFL : Annual Bibliography of Books and Articles on Pedagogy in Foreign 

Languages , Dale L. Lange, May 1970. (ED 040 625) 

Type - B Effort = II Visibility = M Subject = Instr. Content 

Songs in the Foreign Language Classroom , Focus Reports on the Teaching of 
Foreign Languages, #12, Olivia Munoz, September 1969. (ED 034 450) 

Type “ P Effort = L Visibility = M Subject = Instr. Content 

The Mechanical Potential of the Language Laboratory , Focus Reports on the 
Teaching of Foreign Languages, #14, Edward M. Stack, May 1970. (ED 038 072) 

Type =* P Effort = L Visibility = M Subject = Instr. Content 

82. Linguistics and Foreign Language Teaching . ERIC Focus Report on the Teaching 
of Foreign Languages, #21, Freeman Twaddle, December 1970. (ED 044 381) 

T yP e * p Effort = L Visibility = H Subject = Instr. Content 

83 - Directions in Foreign Language Testing . Rebecca M. Valette, 1969. 

(ED 034 460) 

T yP e = R Effort = M Visibility = L Subject = Instr. Content 

E- L E_S: Types of Programs , ERIC Focus Report on the Teaching of Foreign 
Languages, #16, Lester W. McKim, October 1970. (ED 043 268) 

Type * R Effort = L Visibility = M Subject = Instr. Content 

85. New Scheduling Patterns and the Foreign Language Teacher . ERIC Focus 
Report on the Teaching of Foreign Languages, #18, Jermaine D. Arendt, 
November 1970. (ED 043 269) 

Type = R Effort = L Visibility = M Subject - Instr. Content 

LIBRARY AND INFORMATION SCIENCES 

86. ERIC Products 1969-1970 , 1970. (ED 041 598) 

Type = B Effort - H Visibility = M Subject = Ed. Adm. /Services 



* 



87 . The Economics of Information : Bibliography and Commentary on the 

Literature , h.A. Olsen, January 1971. (ED 044 545) 

Type * R Effort « H Visibility ■ L Subject * Ed. Adm. /Services 

88. Library Serials Control Systems; A Literature Review and Bibliography , 

Elizabeth Pan, December 1970. (ED 044 538) 

Type ■ R 'Effort “ M Visibility * L Subject * Ed. Adm. /Services 

READING 

89. Research on Reading: Word Lists , ERIC/CRIER Reading Review Series, 

Bibliography 18, Mary K. Dunn & James L. Laffey, Sept, 1969. (ED 030 778) 

Type ■ B Effort » L Visibility * L Subject * Instr. Content 

90. Research on Elementary Reading; Interest and Tastes , ERIC /CRIER Reading 
Series, Bibliography 29, Chloe Anne Miller (Comp.), August 1970. 

(ED 042 593) 

Type “ B Effort °* H Visibility * L Subject * Instr. Content 

91. Accountability and Performance Contracting , William E. Blanton, November 
1970. 

Type * B Effort * L Visibility * L Subject * Ed. Adm. /Services 

92. Guide to Materials for Reading Instruction, Supplement 1 , Wayne E. Berridge 
and Larry Harris, Sept. 1969. (ED 032 452) 

Typ* ■ P Effort * M Visibility * L Subject * Instr. Content 

93. Reading: What Can Be Measured ? Roger Farr, 1969. (ED 033 258) 

Type «* R Effort ■ H Visibility * H Subject * Instr. Content 

RURAL EDUCATION AND SMALL SCHOOLS 

94. Annotated Bibliography and Descriptive Summary of Dissertations and Theses 
on Rurality and Small Schools , David R. & Tanya S. Kniefel, May 1970. 

(ED 039 962) 

Type ■ B Effort * M Visibility ** L Subject * Spec. /Other Groups 



95. American Indian Education: A Selected Bibliography , Cecilia J. Martinez 

& James E. Heathman, 1969. (ED 030 780) 

Type = B Effort = M Visibility = L Subject = Spec. /Other Groups 

96. School Gardens & Farms — Aspects of Outdoor Education , Peggy Miller, 
December 1970. (ED 045 249) 

Type = P Effort = M* Visibility = M Subject = Instr. Content 

97. Student Activism- An Overview , Janes E. Heathman, Dec. 1970. (ED 045 250) 

Type = P Effort = L Visibility = L Subject = Ed. Adra. /Services 

98. A Synthesis of Current Research in Migrant Education , James 0. Schnur, 
May 1970. (ED 039 049) 

Type = R Effort = M* Visibility = M Subject = Spec. /Other Groups 

99. Education Innovations in Rural America , Alfred P. Wilson, December 1970. 
(ED 045 241) 

Type ~ R Effort = M* Visibility = M Subject * Spec. /Other Groups 

100. The Educational Disadvantage of the Indian American Student , L. Madisan 
Coombs, July 1970. (ED 040 815) 

Type = R Effort = M* Visibility = L Subject * Spec. /Other Groups 



SCIENCE AND MATHEMATICS EDUCATION 

101. Teacher Education, Science Education Information Report, General Biblio- 
graphy Ser i es 2 2, July 1969. (ED 032 441) 

Type = B Effort = L Visibility = M Subject = Instr. Content 

102. Science and Mathematics for Young Children: An Annotated Bibliography , 

“ Science Education Information Reports, Bibliography '1 , Francis Tneiss, 

Sept. 1969. (ED 033 259) 

Type = B Effort = H Visibility = M Subject = Instr. Content 

103. Documents on Science and Mathematics Education in RIE, Science and Math 
Education Information Report, Special Bibliography Series 2 , Cassandra 
Balthaser (ed.) , August 1970. (ED 045 389) 

Type = B Effort = L Visibility = M Subject = Instr. Content 



104 . 



Inservice Education for Teachers of Secondary School Science) Science 
Education Information Reports, Science Paper 1 , Patricia Blosser , September 
1969. (ED 034 912) 

Type * R Effort - M Visibility - M Subject - Ed. Adm./Serv. 

105. Inservice Education for Teachers of Elementary School Science, Science 
Education Information Reports , Patricia Blosser, December, 1969. 

(ED 036 680) 

1yp e a r Effort = M Visibility - M Subject “Ed. Adm./Serv. 

106. A Summary of Research in Science Education for the Years 1963~64 , 

Elementary School Level, Science Education Information Reports , John D. 
Cunningham and David P. Butts, January 1970. (ED 040 304) 

Type * R Effort * M Visibility = M Subject * Instr. Content 

107 . A Summary of Research in Science Education for the Years 1965-67 
Elementary School Level, Research Review Series , Richard Haney , et al . , 
December 1969, (ED 038 554) 

Type = R Effort = M Visibility = M Subject = Instr. Content 

108 . Prograas for Improving Science instruction in the Elementary School- 
Part I, ESS , Robert Rogers & Alan Voelker, January 1970. (ED 039 128) 

Type ■ R Effort - M Visibility * H Subject * Instr. Content 

109. Programs for Improving Science Instruction in the Elementary School , 

Part II, SCIS, Barbara S. Thomson and Alan M. Voelker. (Reprint, 

May 1970) 

Type = R Effort = M Visibility = H Subject * Instr. Content 

110. A Summary of Research in Science Education for the Years 1965-67 , 

Secondary School Level, Science Education Information Reports , Paul 
Westmeyer, et al. , September 1969. 

Type * R Effort = M* Visibility * K* Subject = Instr. Content 
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SOCIAL SCIENCE EDUCATION 



111. Off the African Shelf: An Annotated Bibliography on Society and 

Education , Joanne Binkley, December 1970. (ED 044 349) 

Type = B Effort = M Visibility = L Subject = Instr. Content 

112. Preparing to Teach Economics: Sources and Approaches, Interpretive 

Series #2 , Suzanne Wiggins Helburn, April 1971. (ED 049 997) 

Type = P Effort = M Visibility ■ L Subject = Instr. Content 



TEACHER EDUCATION 

113. A Selected Annotated Bibliography on Differentiated Staffing, Bernard 
McKenna, October 1969. (ED 033 898) 

Type = B Effurt = M Visibility = M Subject = Ed. Adm. /Services 

114. Multicultural Education: A Selected Annotated Bibliograph y, Moira B. 

Mathieson & Rita M. Tatis, September 1970. (ED 043 572) 

Type = B Effort = M Visibility = L Subject = Spec. /Other Groups 

115. Individual Instruction: Part I of a Bibliographic Series on Meeting 

Special Educational Needs , Lorraine Poliakoff, October 1970. 

(ED 044 381) 

Type = B Effort = L Visibility = L Subject = Higher Ed. 

116. Ethnic Groups: Negroes, Spanish Speaking, American Indians and 

Eskimos: Part 4 of a Bibliographic Series on Meeting Special Educational 

Needs, Lorraine Poliakoff, October 1970. (ED 044 384) 

Type = E Effort = L Visibility = L Subject = Spec. /Other Groups 

117. Micro t eaching : History and Present Status , Dwight Allen & James Cooper, 

February ,1970. (ED 036 471) 

Type = R Effort = L Visibility = L Subject = Higher Ed. 

118. A Reader's Guide to the Comprehensive Models for Preparing Elementary 

Teachers , Kaliopee Lanzillotti & Joel L. Burdin (Eds.), December 1969. 

(ED 034 076) 

Type = R Effort = M Visibility = M* Subject - Higher Ed. 

119. Simulation in Preparing School Personnel! ., Donald Cruickshank & Frank 

Broadbent, February 1970. (ED 036 470) 

Type = R Effort = L Visibility = L Subject = Higher Ed. 
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120 . 



Preparing School Personnel for Differentiated Staffing Patterns: A Guide to 

Selected Documents in the ERIC Collection, 1966-1968 , Marlene Ross * May 
1969. (ED 028 155) 

Type * R Effort = L Visibility * L Subject - Ed. Adm. /Services 

121. Classroom Observation Systems in Preparing School Personnel , J. T. Sandefur 
And A. A. Bressler , March 1970. (EJT037 377) 

Type » R Effort = L Visibility = L Subject = Higher Ed. 



TEACHING OF ENGLISH 

122. Basic Annotated Bibliography on Censorship , Lee A. Burress , Jr., January 
1970. 

Type = B Effort * M* Visibility = M Subject - Instr. Content 

123. A Guide to Available Project English Materials (Revised Edition ) 

Donna Butler and Bernard O’Donnell, September 1969. (ED 034 775) 

Type = B Effort « L Visibility * L Subject * Instr. Content 

124. An NCTE/ERIC Report on the Evaluation of Head Start Programs , Robert V. 
Denby, December 1969. (Reprint) 

Type = B Effort = L Visibility * H Subject - Spec. /Other Groups 

125. A Reference Shelf on Curriculum Planning for the Language Arts K-8 , 
NCTE/ERIC Report, Robert V. Denby, March 1970. (Reprint) 

Type = B Effort = L Visibility » H Subjed ’ Instr. Content 

126. NCTE ERIC Summaries & Sources: Film Study at the Secondary Level , 

Robert V. Denby, November 1969. (Reprint) 

Type 3 B Effort = L Visibility ® H Subject * Instr. Content 

127. NCTE/ERIC Suraaaxies and Sources : Inservice Education for Secondary English 

Teachers: ’So Little Time. . .So Much to Learn .’, Robert V. Denby, 

April 1970. (Reprint) 

Type = B Effort = L Visibility = H Subject = Ed. Adm. /Services 

128. NCTE/ERIC Summaries and Sources: Literary Analysis in Secondary English 

Classes , Robert V.^ Denby, March 1970. (Reprint) 

Type = B Effort = L Visibility = H Subject = Instr. Content 
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129 - Poetry Instruction in the Elementary Grades: An NCTE/ERIC Report , 

Robert V. Denby, October l?t>9. (Reprint) 

Type — B Effort = L Visibility = H Subject — Instr. Content 

130* NCTE/ERIC Report: Bilingual Education: A Special Report from CAL/ERIC , 

Anna Maria Malkoc and A. Hood Roberts, May 1970. (Reprint) 

Type = B Effort =L Visibility = H Subject = Spec. /Other Groups 

131. Basic Bibliography on Handwriting, Capitalization, and Punctuation , 

L. Jean York, December 1969. 

Type = B Effort = M * Visibility = M Subject = Instr. Content 

132. Elective English Programs in Junior and Senior High Schools , Linda 
A. Kubicek (Comp.')- September 1970. (ED 041 182) 

Type = P Effort = L Visibility = L Subject = Instr. Content 

133. Recommended English Curriculum Guides K-12 and Criteria for Planning 

and Evaluation : 1970, William J. Scannell, November 1970. (ED 044 419) 

Type = P Effort = L Visibility = H Subject = Instr. Content 

134. Teaching Literature in the Elementary School , NCTE/ERIC Studies in 
the Teaching of English, Norine Odland, July 1969. (ED 031 482) 

Type = R Effort = M* Visibility = M Subject = Instr. Content 

135. Creativity in the English Program , NCTE/ERIC Studies in the Teaching 
of English, Rodney P. Smith, Jr., April 1970. (ED 038 413) 

Type - R Effort = M * Visibility = M Subject = Instr. Content 
TESTS, MEASUREMENT, AND EVALUATION 

136* Developing Criterion-Referenced Tests . Rex Jackson, June 1970. (ED 041 052) 
Type = R Effort = L Visibility = L Subject = Ed. Adm. /Services 
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VOCATIONAL AND TECHNICAL EDUCATION 



137. Work Experience for Broadening Occupational Offerings; A Selected 
Bibliography for Use in Program Development , Information, David 
McCracken, November 1969. (ED 034 062) 

Type « B Effort - M Visibility “ L Subject “ Instr. Content 

138. An Application of Research: Working with Opinion Leaders to 

Accelerate Change in Vocational-Technical Education , Garry R. Bice, 
November 1970. (ED 044 502) 

Type “ P Effort * M Visibility *= M Subject = Ed. Adm. /Services 

139. Review and Synthesis of Research on Vocational Education in Rural Areas , 
B. Eugene Griessman & Kenneth G. Densley, December 1969. (ED 034 632) 

Type » R Effort * M Visibility - L Subject * Instr. Content 

140. Review and Sys thesis of Research: Analysis for Curriculum Development in 
Vocational Education , Milton E. Larson, October 1969. (ED 035 746) 

Type « R Effort - H Visibility *» M Subject « Instr. Content 

141. Review and Synthesis of Research on the Placement and Follow-up of 

Vocational Education Students, Research Series No. 49 , J. Kenneth 
Little, February 1970. (ED 037 543) 

Type = R Effort * M Visibility - M 'Subejct * Ed. Adm. /Services 

142. Review and Synthesis of Research in Trade and Industrial Education , 

Albert J. Paulter and Carl J Schaefer, Sept. 1969. (ED 036 638) 

Type = R Effort =* M Visibility = M Subject = Instr. Content 

Review and Synthesis of Research in Business- and Office Education , Ray G. 
Price & Charles R. Hopkins, April 1970. (ED 038 52^} 

Type * R Effort = H Visibility ■ M Subject = Instr. Content 

Review and Synthesis of Research in Technical Education , Donald S . 
Phillips and Lloyd D. Briggs, October 1969. (ED 036 639) 

Type = R Effort = M Visibility = L Subject = Instr. Content 
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■fe e yi ew and Syn thesis of Research and Developmental Activit ies Concerning 
State Advisory Councils on Vocational Education . Joseph R. Clary 
September 1970. (ED 043 744) 

Type = R Effort = M Visibility = M Subject = Ed. Adm. /Services 

Interpretation of Literature on Career Lddders and Lattices in Health 
Occupations Education , Jean Kintgen, Sept. 1970. (ED 042 919) 

Type = R Effort = L Visibility = L Subject = Instr. Content 
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APPENDIX B 

FORMS USED IN OBTAINING 
LEVEL-OF-EFFORT AND VISIBILITY DATA 



This Appendix contains copies of forms that were used in obtaining level-of- 
effort and visibility data for products in the sample. Differences in programs 
required that three different forms be used. They are: 

1. Form for ERIC clearinghouses and the Educational Materials Center 

2. Form for NCEC staff concerned with the PREP reports 

3. Form for states (in tracing secondary distribution of 
PREP reports) 

One form was completed by the appropriate NCEC unit for each document. 
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For ERIC Clearinghouses and the 
Educational Materials Center 



1 



FOR PRODUCT LISTED AS #1 




i.l 



went into the preparation^? ^his product?* (Pl^s? 1 ?" 31 oan ' hours that 

a h p: h L r:: 

abstracting) and would have been incurred'^whe^ 81 '’ 8 .!/ 1 ' 6 ' ’ lndexln 8 and 
prepared or not. But do include all n r wh ® ther this product was 
specifically to the product, whether on°th SSl ° nal man “ hours devoted 
commissioned authors? or ^ Staff ’ 



1.2 



What is your best estimate of 
clerical man-hours that went 
this product? (Again, please 
incurred in any event, but do 
staff effort.) 



° f ^-professional or 

Pvri if i Caily lnt ° the P re P ar ation of 
exclude man-hours that would have been 

include outside effort as well as 



1.3 



Sfiini professional or 
clerical jnan— hours 



costs of ^IVrlZl^Vo TeJcllZ best ° sUmaCe of the other 
existed? (Please include supplies commit * 5f rSt camera "ready copy 
extent that they were used specif icaUv^n C ° the 

product. Exclude all costs subsequent to the 2 Preparation of this 
copy, such as press-run costs.) existence of a first 



Please identify the types 



Total 

of operations included in 



non-labor costs 
this total 




1.4 



product^that mu^be ttllulllTln t0 the Preparation of this 

labor and non-labor costs above, plllTTplTinTereV^ £Stlmates of 




V 




B-l- 



2 



The next three questions deal with the distribut 4 on of, and dissemination 
strategy for, this document. 



1*5 Please estimate the total number of copies of this product that have 

actually been distributed either by your clearinghouse or by any other 
organization in any form. Exclude EDRS distribution , but please include 
journal circulation statistics, if applicable. 



Total number of copies 

actually distributed 

1.6 If there are special circumstances (e.g., documents were sold or were 
distributed free) related to the distribution of this product that 
should be considered in conjunction with your responses to the 
previous question, please explain: 



1.7 Did this product receive special publicity, other than the usual mention 
in a clearinghouse newsletter or association newsletter? That is, was 
special attention drawn to the product in any of the following ways? 

It was reviewed in the professional literature (please provide full 

journal citation(s). 



It was extensively described — "featured" — in a column, brochure, 
or other publicity medium. 

It received special attention because it was part of a larger 
effort, such as a well-publicized product series or a professional 
gathering, such as a workshop. 

It came to the attention of a large relevant audience because it 
was distributed through a well-established channel such as a 
professional journal. 

It was placed on automatic distribution to individuals on 
our mailing list. 

Other unusual publicity or attention (please specify): 




Ks t 'j 
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1.8 Summing up the preparation and distribution of this product, would you 
say that, relative to the other products of any type that you have 
prepared, the effort invested in it was... 

Just average Above average Below average 



1.9 Would you say that, relative to the other products of any type that 
you have issued, the exposure it achieved was... 

Just average Above average Below average 




B-5 

o i-,. 



2 



. For PREP Reports 
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V 



♦ 



[ Name of PREP Report] 



1. What was the cost of the Targeted Communication contract? 

$ 

2. What is your best estimate of the number of professional man-hours (in USOE} 
that went into the preparation of the: 

a. PREP Report Professional man-hours 

b, PREP Brief Professional man-hours 



3. VThat is your best estimate of the number of USOE semi-professional or 
clerical man-hours that went into the preparation of this PREP report? 



a. 


PREP 


Report 


Semi-professional or 
clerical man-hours 


b. 


PREP 


Brief 


3erai-prof essioml or 
clerical man-hours 



4. In addition to labor costs, what is your best estimate of the other 
costs of this product up to the point when a first camera-ready copy 
existed? (Please include materials, supplies, etc. , to the extent 
that they were used specifically in the preparation of this product. 
Exclude all costs subsequent to the existence of a first copy, such 
as press-run costs.) 



Total non-labor costs 



Please identify the types of operations included in this total. 
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5. If there are special circumstances related to the preparation of this 
PREP report that must be considered in conjunction with your estimates 
of labor and non-labor costs above, please explain here' 



The next three questions deal with the distribution of, and dissemination 
strategy for, this document, from USOE. 



6. Please estimate the total number of copies of this PREP report that hav e 
actually been distributed by USOE to the states. Exclude EDRS 
distribution. 



Total number of PREP 
Briefs distributed 



Total number of PREP 
Reports distributed 



7. If there are special circumstances related to the distribution of this 
Report (or Brief) that should be considered in conjunction with your 
responses to the previous question, please explain: 



8. Did this PREP receive special publicity, other than through the Brief? 
That is, was special attention drawn to the product in any of the 
following ways? 

It was reviewed in the professional literature (please provide 

full journal citation(s). 



It was extensively describea — "featured" — in a column, brochure, 
or other publicity medium. 
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It received special attention because it was part of a larger 
effort, such as a well-publicized product series or a professional 
gathering, such as a workshop. 

Other unusual publicity or attention (please specify) : 



9. Summing up the preparation and distribution of this PREP, would you 
say that, relative to the other PREP’s that you have prepared, the 
effort invested in it was . . • 



Just average Above average Below average 



10. Would you say that, relative to the other PREP's that you have issued, 
the exposure it achieved was . . • 



Just average Above average Below average 
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[Name of PREP Report] 



The following questions. crplo,n the- di 
State in the distribution of the PREP 



ssem tnrt tiou r.tr afegi es used 
Repo r t id c nt i f i ed abov o . 



by your 



1. Did you distribute the brief for this particular TuKT Report? Yes ( ) No ( ) 

If yes, please give an estimate of the total number of copies distributed 
and identify the targeted prof esrional p, roups v/lio were sent copies. 

Estimated number of copies distributed 

Targeted audiences 



2. Did you prepare any special announcement materials concerning the 
availability of this particular PPd'P Report? Yes; ( ) No ( ) 

If yes, please identify the method of distri.bu ti on, estimated number 
of copie!S distributed, and targeted audiences. 

Method of distribution (e.g., brochure, journal or newsletter article) 



Estimated number of copies distributed 
Targeted audiences 



3. How do you make copies of the PREP Reports available to educators in your 
State? 

Copies are reproduced on request. 

Copies are reproduced and automatically distributed to a selected 
audience. 

Copies received from USOE are automatic ally circulated to selected 
staff members. 

Copies received from USOE are made available on-loan, as requested. 
Other (please specify:) 
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As of this date, please provide an est ima tv of the t.o t . ►'< tuehe,' of 
copies distributed by you?: State- for this particular Hdo’ report. 

Total Kos-iboi Copies 



5. 



if there (ire auy special o i i* c*i » ■«» s* i races relate,* to the* tljrtj •■* ? - f en o< 
llrl.v P Keport flint, should he* eonv.iderc d in conjunction \.- H t >ou? 
l espouses t o the previous questions, pi cane to; pi a in : 



In recent: months, have you changed the precede r. £. described above in 
any v:ay due to the avail abi .lily of PUirP's from CTO? 

Yes ( ) No ( ) 



if yes, please explain 
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APPENDIX C 



TABULATIONS ON SEMI-PROFESSIONAL/ 
CLERICAL MAN-HOURS AND 
NON-LABOR COSTS 



The following three tables represent data obtained from NCEC units on three 
cost components: 

• Professional Man-Hours 

• Semi-Prof essional/Cler ical Man-Hours 

• Non- Labor Costs (in dollars) 

"Professional-man-hour" units were used to create the level-of-ef fort indexes 
for each product type. The distribution of the other two cost units are provided 
to show the relationships between types of costs. 

All 146 documents are not represented in these tables. A document with only 
partial data^ or no data at all, was assigned the median number of hours for 
the appropriate product type in the level-of-ef fort indexes. These data are 

presented only as general information; they are not keyed to actual document 
numbers. 
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TABLE C-l. DISTRIBUTION OF COSTS FOR INDIVIDUAL REVIEWS 
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All Practical Guidance Papers Iti this index level are PREP Reports. Professional Man-Hours are presented in estimates of the total 
Targeted Communication contract or grant, plus the hours in NCEC for the preparation of the actual PREP Reports. The second two 
columns are direct PREP-report costs only — not contract-related costs. 

**These data represent four individually packaged PREP reports on a particular topic. In the General Field Survey, they. were treated 

as a single document. (See Chapter III, page 8.) 





TABLE C-3 . DISTRIBUTION OF COSTS FOR INDIVIDUAL BIBLIOGRAPHIES 













‘X- 

X 



) 









CJ 








o 

ERIC 



*• ^ 



M 

O ** 0 . 



* 



< 

> 



k* 

^ 3 ; 






APPENDIX D 

QUESTIONNAIRE PACKAGES 



This appendix includes copies of each of the four survey instruments: 

1. Screening Questionnaire (Ql) 

2. User Evaluation Questionnaire (Q2) 

a. Version A: for Reviews and Practical Guidance Paper 

b. Version B: for Bibliographies 

3. Non-user Evaluation Questionnaire (Q3) 

4. Specialists' Questionnaire (Q4) 

Cover letters and followup letters are also enclosed. 



1. Screening Questionnaire (Ql) 



O 

ERIC 



i, u . 



Office of Managamtnt and Budget No. 51 S-71045 
Expiration Date: April 30, 1972 



SYSTEM DEVELOPMENT CORPORATION 

5827 Columbia Pike, Falls Church, Virginia 22041 



January 14, 1972 



Dear Educator: 



Under sponsorship of the U.S. Office of Education (USOE), the System 
Development Corporation is conducting a study of information products prepared 
by USOE's National Center for Educational Communication (NCEC). The goal is to 
assess the quality and utility of these products so that NCEC can better meet the 
needs of the educational community. 

Our first step in the study is to determine the general level of awareness of NCEC 
products among edupators throughout the country. Your assistance in this effort 
will be of great benefit, and we ask your cooperation in answering the four questions 
that follow. For your convenience, a prepaid return envelope is enclosed. We would 
greatly appreciate your returning the completed questionnaire within 10 days. 

Thank you for your help. 



Sincerely, 




Judith Wanger 
Project Director 

NCEC Product Evaluation Project 



1 . in what institutional setting are you working? (Check as many as apply) 



SCHOOLS AND COLLEGES 

( ) Kindergarten or preschool 

( ) Elementary school 

{ ) Junior high or middle school 

( ) Senior high school 

( ) Vocational high school 

( ) Junior college or community college 

( ) Four-year college/university 



OTHER SETTINGS 

( ) School district central office 

( ) State department of education 

( ) Federal government agency 

( ) Non-profit organization 

( ) Professional society or association 

{ ) Commercial organization 

( ) Other (Please specify) 



2. What is your main professional role or function in the educational community? (If you have a dual assignment, please identify your 
principal role by placing a "1" before the single applicable item below and a " 2 " before the secondary role.) 

{ ) Teacher/Professor ( ) Program specialist, consultant, or coordinator 



( ) Reading specialist ( 

( ) Superintendent or assistant superintendent ( 

( ) President or vice-president ( 

( ) Principal or assistant principal ( 

( ) Supervisor of instruction ( 

( ) Academic dean ( 



( ) Student personnel worker or guidance counselor 



) Psychologist 

} Media coordinator or instructional technology specialist 
) Librarian or instructional resources specialist 
) Researcher or R&D staff member 
) School board member 
) Other (Please specify) 



O 

ERIC 



r 
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3 Which of the following areas represent your major professional interests? (Check as many as apply) 



( ) Adult and Continuing Education 

( ) Adult Basic Education 

( ) American Indian Education 

( ) Bilingual Education 

( ) Compensatory Education 

( ) Counseling and Student Personnel Services 

( ) Curriculum Development 

{ ) Early Childhood Education 

( } Educational Administration 

( ) Educational Media and Technology 

( ) Elementary Education 

( ) English and Language Art' 

( } Ethnic and Minority Group Education 

( ) Exceptional Children (gifted, mentally retarded, 

physically handicapped, emotionally disturbed) 

( ) Fine Arts (music, art, drama, dance) 

( ) Health, Safety, and Driver Education 

( ) Higher Education 

( ) Home Economics 



( ) Instructional Materials 

( ) Junior Colleges 

( ) Languages and Linguistics 

( } Library and Information Sciences 

( ) Mathematics Education 

( ) Physical Education 

( ) Psychological Services 

( ) Reading 

( } Research and Development 

( ) Rural Education and Small Schools 

( ) Science Education 

( ) Secondary Education 

( ) Social Science and Social Studies Education 

( ) Teacher Education 

( ) Tests, Measurements, and Evaluation 

( ) Vocational and Technical Education 

( ) Other (Please specify) 



NOTE: 



Please circle the one area that represents your principal professional interest. 
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This final question concerns your famtliirity with the information products of the U.S. Office of Education's Nation al Center for 
Educational Communication (NCEC). 

Three NCEC units are currently involved in synthesizing educational liter ature 

The Division of Practice Improvement prepares interpretive summaries, culled PREP Reports Putting Research uUo Educational 
Practice- which are disseminated through State education agencies, the ERIC Document Reproduction Service (EDFiS). and the 
Government Pointing Office 

The Educational Mate* u* ; s n ter (formerly known as the Educations! Materials Laboratory} piepares bibliographies covering 
textbooks, children s booNt, and professional education materials that aie made available upon request fioni the Center, through 
the Government Printing Office and EDRS 

The 19 ERIC clearinghouses produce many kinds of products, including state of the art reports, bibliographies, and handbooks. 
These documents are distributed through clearinghouses, professional associations, and EDRS. 

The enclosed insert displays miniature examples of these NCEC products. They are provided to assist you in answering the 
following question. 

4. Prior to receiving the questionnaire, had you seen or read an information product prepared by any one of these NCEC units? 
(Check as many as apply) 



a. 


NCEC UNITS 

DIVISION OF PRACTICE IMPROVEMENT - PREP REPORTS 


1 have read 
or skimmed 

( ) 


1 ?m aware 
of, but have 
not read or 
skimmed 

( ) 


l am not 
aware of 

( ) 


b. 


EDUCATIONAL MATERIALS CENTER - BIBLIOGRAPHIES 


( 


) 


( 


) 


( 


I 


c. 


ERIC (Educational Resources Information Center) 
PRODUCTS, from the clearinghouses on: 

Adult Education (ERIC/AE), Syracuse University 


( 


> 


( 


) 


( 


) 


d. 


Counseling and Personnel Services (ERIC/CAPS), Ann Arbor, Michigan 


( 


) 


( 


) 


( 


) 


e. 


Disadvantaged (ERIC/IRCD), Columbia University 


( 


) 


( 


) 


( 


) 


f. 


Early Childhood Education (ERIC/ECE), University of Illinois 


( 


) 


( 


) 


( 


) 


9- 


Educational Management (f£RIC/CEA), University of Oregon 


( 


) 


( 


) 


( 


) 


h. 


Educational Media and Technology, Stanford University 


( 


) 


( 


) 


( 


) 


i. 


Exceptional Children (ERiC/CEC), The Council for Exceptional 
Children, Arlington, Virginia 


( 


) 


( 


) 


( 


) 


J. 


Higher Education (ERIC/CHE), George Washington University, 
Washington, D.C. 


( 


) 


( 


) 


( 


) 


k. 


Junior Colleges, University of California at Los Angeles 


( 


) 


( 


) 


( 


) 


L 


Languages and Linguistics (MLA/ERIC), Modern Language Association 
of America, New York, New York 


( 


) 


( 


) 


( 


) 


m. 


Library and Information Sciences (ERIC/CLIS), American Society for 
Information Sciences, Washington, D.C. 


( 


) 


( 


) 


i 

% 


) 


n. 


Reading (ERIC/CRIER), Indiana University 


( 


) 


( 


) 


( 


) 


o. 


Rural Education and Small Schools (ERIC/CRESS), New Mexico State 
University 


( 


) 


( 


) 


( 


} 


p. 


Science and Mathematics Education (ERIC/SMAC), Ohio State 
University 


( 


) 


( 


) 


( 


) 


q. 


Social Science Education (ERIC/CHESS), Boulder, Colorado 


< 


) 


( 


) 


( 


) 


r. 


Teacher Education, American Association of Colleges for Teacher 
Education, Washington, D.C. 


( 


) 


( 


) 


( 


) 


s. 


Teaching of English (NCTE/ERIC), National Council of Teachers of 
English, Urbana, Illinois 


( 


) 


( 


) 


( 


) 


t. 


Tests, Measurement, and Evaluation (ERIC/TM), Educational Testing 
Service, Princeton, New Jersey 


( 


) 


( 


V 

t 


( 


) 


u. 


Vocational and Technical Education, Ohio State University, Columbus, 
Ohio 


( 


) 


( 


) 


( 


) 



Thank you for your cooperation. 



EXAMPLES OF NCEC INFORMATION PRODUCTS 



The following 26 miniatures are displayed to assist you in answering question 4. 



PREP Reports and Educational Materials Center Bibliographies. . . 



PEBPI 



a. 
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PREP Report (Cover/Fofder) 



b. Educational Materials Center Bibliography 
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c. Adult Education 
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d. Counseling and Personnel Services c ‘ Disadvantaged 




j. Higher Education 
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Focus Reports on 
the Teaching of 
Foreign Language* 




I. Languages & Linguistics 



k. Junior Colleges 
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s. Teaching of English 




q. Social Science Education 



r. Teacher Education 
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f. Early Chile 




m. Library & 
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g. Educations) Management 



h. Educational Media & Technology 







i. Exceptional Children 





Information products from the ERfC cfeeri and. . . 
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NCEC products, as available f r c>m ERIC Document Reproduction Service 




EDUCATIONAL RESOURCES 
INFOIHATIOK CENTER 



ERIC: 
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EDRS Microfiche 
(prior to 1971) 



EDRS Hard Copy 

(format prior to 1971) 





EDRS Hard Copy EDRS Microfiche 

(format since 1971) (since 1971) 
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SYSTEM DEVELOPMENT CORPORATION 

5827 Columbia Pike, Falls Church, Virginia 22041 



February, 1972 



u 



Dear Educator: 

Since we hate not yet heard from you with regard to our NCEC Product 
Evaluation Project questionnaire, I am again appealing to you for 
assistance. As you may remember, System Development Corporation is 
conducting a study of information products prepared by the National 
Center for Educational Communication (NCEC) in the U.S. Office of 
Education. The goal is to assess the quality and utility of these 
products so that NCEC can better meet the needs of the educational 
community . 

Our first step in the study is to determine the general level of 
awareness of NCEC products among educators throughout the country. 

Your assistance in this effort will be of great benefit. 

We realize that you may have only recently returned a completed 
questionnaire; if this is so, please accept our apologies for sending 
you this follow-up letter and thank you for helping us. If you have 
not completed the questionnaire, we would appreciate your taking the 
time to do so now. In the event that you have either lost or misplaced 
the questionnaire, we have enclosed a second copy. 

For your convenience, a prepaid return envelope is enclosed. We 
would greatly appreciate your returning the completed questionnaire 
within 10 days. 

Thank you. 



JW/lsr 

Enclosure 



Sincerely, 




Project Director 

NCEC Product Evaluation Project 



ERIC 



2. User Evaluation Questionnaire (Q2) 

A. for, Reviews and Practical 
Guidance Papers 
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SYSTEM DEVELOPMENT CORPORATION 

5827 Columbia Pike, Falls Church, Virginia 22041 



February, 1972 



Dear Educator: 

As indicated in the letter you recently received from the U.S. Office of 
Education, System Development Corporation is undertaking the second phase 
of its NCEC Product Evaluation Study. Response to the first questionnaire 
was particularly gratifying, and we hope to enlist the continued support 
of several thousand educators in this next step. Because of your knowledge 
of NCEC products, we are inviting you to be one of our product evaluators. 

We greatly appreciate your contribution to the first phase of our study, 
and we hope you are willing to help us again now. 

You are participating in this study with many of your colleagues throughout 
the country; however, each of you brings to this study a unique combination 
of educational responsibilities, professional Interests, and Information 
needs. The success of this project depends upon our obtaining such broadly 
based evaluations. Thus, your assistance is of the utmost importance to us. 

The enclosed evaluation packet contains questions about the quality and 
utility of 10 NCEC products. We are not sending the complete documents . 
to you, just a representation of each — that is, the title page, table of 
contents, and an abstract or extract — to assist r ou in recalling them. 
Further instructions are provided on the cover page of each questionnaire. 

In our data analysis, we will use only code numbers to identify respondents, 
so that evaluations can be held in strictest confidence. A pre-paid return 
envelope is enclosed for your convenience. We would greatly appreciate your 
returning the completed packet within 10 days. 

Thank you for your interest and help. 



Sincerely, 



JW/lsr 
Enclosure 



Project Director 

NCEC Product Evaluation Project 




o 
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^ •-*»' Offk* of Mwwflomont and Budftt No. 51-S-71046 
Expiration Data: April 30, 1972 



NCEC PRODUCT EVALUATION SURVEY 
DOCUMENT NUMBER 



This evaluation packet contains a separate questionnaire and document representation (title 
page, table of contents, and an abstract or extract) for each of 10 documents we are asking 
you to review. The documents are undoubtedly of varying interest to you, and we do not 
expect you to have read all of them. 

The following steps should assist you in this review: 

1. Look at the document representation to the left of the questionnaire to determine if 
you have previously read or skimmed the full document. 

2. Answer question number 1 and, on the basis of this response, then answer the other 
appropriate questions. 



Please return the completed questionnaires. in the enclosed pre-paid return envelope within 
todays. 



Thank you for your cooperation. 



Judith Wanger, Project Director 
System Development Corporation 
5827 Columbia Pike 
Falls Church, Virginia 22041 



□ Check hero if you would like to receive e summery report of this study. 




♦ 
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PREPARING TC TEACH 
ECONOMICS : SOURCES AND 

APPROACHES 

by 

Suzanne Wiggins Helburn 
Interpretive Series No. 2 



Published Jointly by: 

ERIC Clearinghouse for Social Studies/Social Science Education, 

Social Science Education Consortium, Inc. 

970 Aurora 

Boulder, Colorado 80302 

APR 1371 

[drawn from Title Page] 



PREPARING TO TEACH ECONOMICS: SOURCES AND APPROACHES 

This is a "tip sheet" for teachers who ask ERIC/ChESS for help 
in setting up a high school course in economics for the first time. 

It provides these kinds of help: (1) a brief overview of what eco- 

nomics is; (2) suggestions about how economics fits into the high 
school social studies curriculum; (3) references to a few books 
which are particularly good introductions to economics; (4) some 
suggestions about organizing the course; (5) a list of agencies to 
contact for help or for useful student materials; and (6) a short bib 
liography of reference and library materials you may want to order. 
For those readers who already have a grasp of the nature and signi- 
ficance of economics, the next section may not be very useful. 

Feel free to skip it and go directly to Section 2, if this is the 
case . 

[extract from Introduction] 



[taken from Chapter I] 
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FIGURE 1 

THE ECONOMIC PROCESS 




Office of Management and Sudgat No. 51-S-71046 
Expiration Data: April 30, 1972 



NCEC PRODUCT EVALUATION SURVEY 



P(mn look at tha document representation on the (aft and indicata your previous awa r e n es s of the full 
document 

1. ( } I have not previously seen or read this document. (Answer questions 2*3 only.) 

( ) I have previously seen or heard about this document but have not read or skimmed it. (Answer 
questions 2-4 only.) 

( ) I have previously read or skimmed this document. (Skip to question 5 on the next page.) 



The following questions concern your potential interest or lack of interest in this document. 

2. How relevant do you think this document might be to your general professional interests? 

Relevant { ) Somewhat relevant ( ) Not at all relevant ( ) 

3. As of the publication date, how great was your need for a good documant of this type, on this 
topic? 

( ) Document would probably have been very useful. 

( ) Document would probably have bean of some use. 

( ) Document would probably have been of little or no use. 

4. If you knew about tha document but did not read or skim it, what reason (s) do you remember? 

( ) I could not readily obtain a copy. 

% 

( ) I was not sufficiently interested. 

( ) Other (please specify) . 



PLEASE PROCEED TO THE NEXT DOCUMENT REPRESENTATION AND SET OF QUESTIONS. 



ERIC 




Pfeaae unMr questions 5-13 only if you have reed or skimmed the document. 

5. As nearly as you can remember, how recently did you read or skim the document? 

( 1 Within the past month ( } Within the past 6 months 

( ) Within the past 3 months { ) More than 6 months ago 

6. How relevant was the topic to your general professional interests? 

( ) Relevant { ) Somewhat relevant ( ) Not at all relevant 

7. As of the publication date, how great was your need for a good document on this topic? 

( ) Very great; I had an immediate need for a document on this topic. 

( ) Moderately great; the topic is of continuing importance to me. 

( ) Not at ail great; f had no special need for it. 

8. For your needs, how well did the document cover the topic(s)? 

( ) Very well ( ) Moderately well ( ) Poorly 

9. Do you feel that the material was up-to-date in its coverage of current research or practice, as of 
its publication date? 

{ ) Yes { ) No ( ) Could not Judge 

10. The questions below concern various aspects of the document's quality. 

The document was: 

I ) too long ( ) too short ( ) about right 



The organization was: 

( ) excellent ( ) satisfactory ( ) less than satisfactory 



The writing was: 

( ) very clear ( ) moderately clear ( ) hard to follow at timesjr 



The format (physical layout, illustrations, typography, etc.): 

( ) was very helpful ( I did not contribute ( ) hindered readability 

to readability and to readability and and understanding 

understanding understanding 

The discussion was: 

( ) very thoughtful ( ) reasonably thoughtful ( ) inadequate for my purposes 
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' 11. Documents can serve a variety of purposes and fulfill many different information needs for reactors. 

In the following question, please indicate how useful the document was to you for each of the 
purposes listed. If you did not use the document for a stated purpose, check the last column. 

DEGREE OF USEFULNESS 

Somewhat Notat Did not usa for 

PURPOSE OF USE Vary useful useful all useful this purpose 


To obtain an overview 
of the topic 


( ) 


( ) 


( ) 


( ) 


To look up needed facts 


( ) 


( ) 


( ) 


( ) 


To identify individuals 
working in the area 


( ) 


( ) 


( ) 


( ) 


To identify relevant literature 
references 


( ) 

» 


( ) 


( ) 


( ) 


To update my knowledge about a 
subject already known to me 


( ) 


( ) 


( ) 


( I 


To obtain knowledge about a new 
subject 


( ) 


( ) 


( ) 


( ) 


Other (please specify) 


- ( ) 


( ) 


( ) 


( ) 



12. As a result of reading the document, did you use the information or the document in any of the 
following ways? 

( ) Used the facts or recommendations to make a decision 
( ) Applied the information in my own work 

{ ) Used the information to give advice to other people 
( ) Examined other documents 

( ) Consulted with author(s) or other persons identified in the document 
( ) Passed the document on to a coileague(s) to read 
If this documented any special effect on your work, please describe the kind of effect it had. 
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13. In general, how would you compare this document with other documents of the same type? 

( ) ft is a very useful document. 

{ ) It is not unusually useful, but it is worth having available. 

( ) Its usefulness is too limited to justify its publication. 

Please use the space provided below for any suggestions you have concerning the future preparation 
of documents of this type, or for elaboration on any of your responses above. 



PLEASE PROCEED TO THE NEXT DOCUMENT REPRESENTATION AND SET OF QUESTIONS. 
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2. User Evaluation Questionnaire (Q2) 
B. for Bibliographies 
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Offtc* of MwH m m it and Budget No. 
Expiration Data: Ap~*t 30, 1972 



NCEC PRODUCT EVALUATION SURVEY 
DOCUMENT NUMBER 



This evaluation packet contains a separate questionnaire and document itforesentation (title 
page, table of contents, and an abstract or extract) for each of 10 documents we are asking 
you to review. The documents are undoubtedly of varying interest to you, and we do not 
expect you to have read all of them. 

The following steps should assist you in this review: 

1 . Look at the document representation to the left of the questionnaire to determine if 
you have previously read or skimmed the full document. 

2. Answer question number 1 and, on the basis of this response, then answer the other 
appropriate questions. 



Please return the completed questionnaires in the enclosed pre-paid return envelope within 
10 days. 



Thank you for your cooperation. 



Judith Wanger, Project Director 
System Development Corporation 
5827 Columbia Pike 
Palis Church, Virginia 22041 



□ Check hare if you would like to receive a summary report of this study. 
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[BB 005 ] 



LITERATURE OF 
THE PROFESSION 



/t Bibliography Based on the Acquisitions of the Educational Materials Center 
from January 1, 1969 through June 30, 1970 



Compiled by: Eunice von Ende 
Biological Sciences Communication Project 
George Washington University 



[drawn from Cover and Titla Page] 



July 1970 



[partial display] 



CONTENTS 



Page 



l 



3 

5 



6 

6 



I Books and Monographs 

Education— History, Theory, and Issues 
Comparative and International Education 
Educational Psychology 
Educational Research 

Educational Sociology and Related Factors 
Child Development ? 

Guidance and Personnel Services 8 

Home-School-Community Relations 8 

School Organization and Administration . 9 

Curriculum Development 1® 

Educational Facilities 
Educational Media and Technology 

Rural and Urban Education 

Support for Education 

Education of Special Groups 



Page 

Teacher Education 18 

The l eaching of Arts and Crafts 19 

The Teaching of English 20 

General language and Composition 20 



Reading and Literature 

The Teaching of Foreign Languages 

The Teaching of Health, Physical Education, 



21 

22 



10 

11 



It 

13 



Safety Education, and Recreation 
The Teaching of Science and Mathematics 
The Teaching of Social Studies 
Vocational and Technical Education 

General 

Manpower Development and Training 
Technical Education Programs 



22 

23 

24 
24 

24 

25 
25 
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Libraries, Media Centers, and Information 
Services 



25 



I. Books and Monographs 



[sample entries] 



EDUCATION— HISTORY, THEORY, AND ISSUES 



Abelcs, Elvin. The Student and the University: A 
Background Book on the Campus Revolt. Parents, 
1969. 246 pp. 



Cassidy, Harold G. Knowledge , Experience, and Ac- 
tion: An Essay on Education. T. C., Columbia, 
1969. 205 pp. 



American Education: Its Men, Ideas and Institutions. 
Reprint editions. Arno, 1969. 

vCx 



Clarizio, Harvey F., editor. Mental Health and the 
Educative Process: Selected Readings . Rand Mc- 






Office of Management and Budget No. 51-S-71Q46 
Expiration Date: Anril 30, 1972 



NCEC PRODUCT EVALUATION SURVEY 

Please look at the document representation on the left and indicate your previous awareness of the 
full document. 

1 . ( ) I have not previously seen or used this document. (Answer questions 2-3 only.) 

( ) I have previously seen or heard about this document but have not used it. (Answer questions 

2-4 only.) 

( ) 1 have previously used this document. (Skip to question 5 on the next page.) 



The following questions concern your potential interest or lack of interest in this document. 

2. How relevant do you think this document might be to your general professional interests? 

( ) Relevant ( ) Somewhat relevant ( ) Not at all relevant 

3. As of the publication date, how great was your need for a good document of this type, on this 
topic? 

( ) Document would probably have been very useful. 

( ) Document would probably have been of some use. 

( ) Document would probably have been of little or no use. 

4. If you knew about the document but did not use it, what reason(s) do you remember? 

( ) I could not readily obtain a copy. 

( ) I was not sufficiently interested. 

( ) Other (please specify) 



PLEASE PROCEED TO THE NEXT DOCUMENT REPRESENTATION AND SET OF QUESTIONS. 



t 






% 






Please answer questions 5-13 only if you have used tJ a document. 

5. As nearly as you can remember, how recently did you use the document? 

* 

{ ) Within the past month ( ) Within the past 6 months 

( ) Within the past 3 months ( ) More than 6 months ago 

6. HoW*^J® vant was the topic t0 your 9® nera ‘ Professional interests? 

( ) Relevant ( ) Somewhat relevant ( ) Not at all relevant 

7. As of the publication date, how great was your need for a good document on this topic? 

{ ) Very great; 4 had an immediate. need for a document on this topic. 

( ) Moderately great; die topjb is of continuing importance to me. 

- t * • • < 

4 . * ' 

'{, ) Not at all great; I had no special need for it. 

_ ? 

8. For your needs, how well did the document cover the topic(s)? 

( ) Very well ( ) Moderately well ( ) Poorly 

9. Do you feel that the material was up-to-date in its coverage of current research or practice, as of 
its publication dcte? 

{ ) Yes ( } No ( ) Could not judge 

10. Thjijiuestions below concern various aspects of the document's quality and utility. 

The number of references was: . 

too many ( ) too few ( ) about right 

The classification or organization of entries {references) was: 

{ ) excellent ( ) satisfactory ( ) less than satisfactory 
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{ ) hindered use 



The format (physical layout and typography): 

(^^was very helpful ( ) did not contribute 

to its usability to its usability 

The textual material (annotations, abstracts, summaries, etc.) was: 

( } very useful ( ) moderately useful ( ) inadequate for my purposes 



0 
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1 1 . Bibliographies can serve a variety of purposes and fulfill many different information needs for 
users, in the following question, please indicate how you used the document. (Check as many as 

apply) 

( } To identify documents related to particular topics 

( ) To identify documents on particular projects 

( ) To identify documents by particular individuals 

( ) To identify documents from particular institutions 

( ) To perform a comprehensive search of the literature 
( ) To see what kind of new «vork,is being reported 

12. As a result of using this document, did you examine any of the documents cited? 

{ ) Yes ( ) No 

If yes, was the content of the document what you had been led to expect by the content of the 
bibliographic reference? 

( ) Yes ( ) No 

13. In general, how would you compare this document with other documents of the same type? 

( } it is a very useful document. 

( ) It is not unusually useful, but it is worth having available. 

( ) Its usefulness is too limited to justify its publication. 

Please use the space provided below for any suggestions you have concerning the future 
preparation of documents of this type, or for elaboration on any of your responses above. 



PLEASE PROCEED TO THE NEXT DOCUMENT REPRESENTATION AND QUESTIONNAIRE. 
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3. Non-user Evaluation Questionnaire (Q3) 









SYSTEM DEVELOPMENT CORPORATION 

5027 Columbia Pike, Falls Church, Virginia 22041 



February, 1972 



Dear Educator: 

As indicated in the letter you recently received from the U. S. Office of 
Education, System Development Corporation is undertaking the second phase 
of its NCEC Product Evaluation Study. Response to the first questionnaire 
was particularly gratifying, and we hope to enlist the continued support 
of several thousand educators in the next step. On our earlier question- 
naire, you indicated that you had not used NCEC products such as PREP Reports, 
Educational Materials Center bibliographies, or ERIC products. Nevertheless, 
we are very much interested in your views on the potential utility of these 
kinds of products. We hope you are willing to help uc again now. 

You are participating in this study with many of your colleagues throughout 
the country; however, each of you brings to this study a unique combination 
of educational responsibilities, professional Interests, and information 
needs. The success of this project depends upon our obtaining such broadly 
based evaluations. Thus, your assistance is of the utmost importance to us. 

The enclosed evaluation packet contains questions about the quality and 
utility of 10 NCEC products. We are not sending the complete documents 
to you, just a representation of each — that is, the title page, table of 
contents, and an abstract or extract — to assist you in completing the 
questions. Further instructions are provided on the cover page of each 
questionnaire. 

In our data analysis, we will use only code numbers to identif ^ respondents, 
so that evaluations can be held in strictest confidence. A pre-paid return 
envelope is enclosed for your convenience. We would greatly appreciate 
your returning the completed packet within 10 days. 

Thank you for your interest and help. 



Sincerely, 



JW/lsr 




Project Director 

NCEC Product Evaluation Project 



Enclosures 






NCEC PRODUCT EVALUATION SURVEY 
DOCUMENT NUMBER 



o - 
ERIC-: 



This evaluation packet contains • separate questionnaire and document representation (tWa 
page, table of contents, and an abstract or extract) for each of lO d oc um e n t s we washing 
you to review. 

We realize that you will not have read the d oc um en t s; however, we are most interested in 



a range of topics we judged to be of varying interest to you. 

. ' • f 

The following steps should assist you in this review: 

1. 



the questionnaire. 

2. As you 

(e.g., a bibliography, review of res e arch . 




Please return the 

todays. 

Thank you for your c o oper a tion. 
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a handbook 
for teachers of english 
to non-english speaking adults 

* by patricia hefferman- cabrera 



eric 



o 

ERIC 



clearinghouse for linguistics 
clearinghouse on adult education 

[full display] TABLE OF CONTENTS 
Introduction 

1. The WHY and the WHO of TESOL Teaching . . . 



2. Some HOW'S of TESOL Teaching 

3. Some WHAT’S to Teach. . • • 

4. Some KINDS of Evaluation . 

I BIBLIOGRAPHY 



[October 1969] 

[drawn from Title Page] 
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The purpose of this publication Is to pull together Ideas, procedures, 
opinions, and materials which have evolved out of one person's experiences of 
the last ten years, encompassing such experiences as TESOL (Teaching English 
to Speakers of Other Languages) on television to several thousand Cuban 
refugees In Miami, coordinating the Instructional programming for several 
thousand more non-English speakers in adult education programs, serving as the 
consultant to the California State Department of Education on matter, of 
TESOL for Mexlcan-American adults and developing and teaching the Certificate 
and Degree Program for TESOL teachers at the University of Southern California 
It ls my 0 pmion that the TESOL teacher should know about each technique 

and method and should be able to evaluate materials, thereby selecting and 
developing a style of teaching compatible with the expectations and needs of 
his students. Therefore, we shall think through such topics a. who teaches 
TESOL and who is taught, how do we teach it, what do we teach, and how can we 

evaluate what ve have taught. 
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NCEC PRODUCT EVALUATION SURVEY 



PImm look it th# dooufMftt unriiiiif tion on # m Mt and indtoiti vour oMioin 

w wo n*»w mwvvtvwi 1 pi^f wavv^wwv iw vor i^p»o wto y wow wop^ow 



1 . Have you previously road or skimmed this document? 
Yes ( ) 




2a. Had you raad or h oard about this document prior to 
learning about this survoy? 

I > Yes ( ) No “ 



3a. How relevant was this document to your general 
professional interests? 

( ) Relevant 

A ) Somewhat relevant 



( ) Not at all relevant 



4a. How useful wee this document to you? 



( ) Very useful 
( ) Somewhat useful 



( ) Not at all useful 



No | ) 




2b. How relevant do you think this document might be to 



I ) R el evant 

* , r* < 

I pkh at an rmnnt 

3b. As of the pubSoetfon date, how frees wee your need 

*vi i 0OwO iWOWWire Vy B i i wR xsm 10^9£s 



( ) Document would 
ustfcd. 



pmoaoty new noon very 



I ) Document ssould orcbabfv have been of oome 

offfttk 



4b. 




If 

^ y ^ p s _ f l i 

f y 1 was not sufficiently IMWesied. 
J uroryiii yiCffYi -.. , 




*WN 



n m# 



Me ■■ 



* ■ 4* . 



PLEASE PROCEED TOTHENE^CT DOCUMENT R&REtGVTATiaM AN0SCTOF QUEfTOf*. 
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Specialists' Questionnaire (Q4) 
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SYSTEM DEVELOPMENT CORPORATION 

C (jlmnbfti FM k t % C fum !:, ViHjima 2 JIM I 

(703) 820-2220 



February 16, 1972 



Dear Specialist: 

We are most pleased to have you as a participant in the Specialists' Survey 
of the USOE-sponsored NCEC Product Evaluation Project. Enclosed you will 
find the following items: 

1. Document(s) . As we Indicated in our telephone conversation, 
documents have been assigned in the subject areas for which 
you were recommended by ERIC Clearinghouse Advisory Board 
members. If you do not feel that a document is truly relevant 
to your general professional interests, please indicate this on 
the appropriate question in the attached questionnaire, and do 
not complete the evaluation. We would appreciate your contacting 
us about this problem as soon as possible, so that we can re- 
assign the aocument(s) to another specialist. 

2. Questionnaire . We have attached a 4-page questionnaire to each 
document we are asking you to review. In addition to the check- 
mark responses, we hope you will take the time to complete the 
open-ended sections, so that we may have concrete examples and 
suggestions to support recommendations made in our report to USOE. 

3. Remuneration Form . Attached to your first questionnaire is a 

Remuneration Form. We would appreciate your completing this form, 
as it will help us expedite the paperwork that accompanies this 
exchange . 



A pre-paid return envelope is enclosed for your convenience. You are 
welcome to the documents, and we ask that you return only the completed 
questionnaire(s) and Remuneration Form. We hope that you will be able 
to complete the review by March 10. 

We are encouraged by the support and enthusiasm expressed for the study 
during our telephone conversations with you and our other specialists. We 
look forward to receiving your evaluation and, in turn, will send you a 
summary report of the study. Thank you again. 



JW/ 1 sr 

Enclosure 





I'* v>j ^ l L) i recto i 

NCEC Product Evaluation Project 
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NCEC PRODUCT EVALUATION STUDY 



Remuneration Form 
for 

Specialists’ Survey 



SDC has asked me to evaluate the following NCEC products: 

SDC Code Number Evaluation Completed 

(Please check) 



1 . 



2 . 



3. 









4. 



5. 



6 . 



I understand that I am to receive $15.00 for each document evaluation 
I have checked above. 

1 1 Please check here it you are not eligible to accept remuneration 
from a federally sponsored project. 

Name 



Position 

Institutional 

Affiliation 

.V it ess 



(City) (State) (Zip'; 




Q O 

O i ** 



Office of Management and Budget No. 51-S-7tQ46 
Expiraton Date: April 30, 1972 



t 



NCEC PRODUCT EVALUATION SURVEY 



1. Were you involved in any way in the preparation of this dc 't 'merit? (check one) 



i 



( ) Was not involved 



( } Reviewed it (pre-publication) 



( ) Authored or co-authored it ( ) Reviewed it (post-publication) 



( } Edited it 



( } Other folease specify) 



a,. A j».~ 



If you authored, edited, or reviewed the document before publication, skip to the next 
Otherwise, continue with question 2. 




4 



2. How relevant is this document to your general professional interests? 



4 



Relevant ( ) Somewhat relevant ( ) Not at all relevant ( ) 






if this document is not at ail relevant to your general professional interests, skip to the next document. 
Otherwise, continue with question 3. 

3. Had you read this document previously? 

Yes ( ) No ( ) 

If yes, when did you read it? 

( ) Within the past month ( ) Within the past 6 months 

( ) Within the past 3 months ( ) More than 6 months ago 



Each of the following questions addresses some dimension of quality or utility. Answer each one with 
respect to your knowledge of the field and the needs of professionals in the field, as you understand them 
to be. 

4. As of the publication date, how great was the need in the field for a good document of this type on 
this topic? 

( ) Very great; there was an immediate need for a document on this topic. 

( ) Moderately great; the topic is of continuing importance in the field. 

( ) Not at all great; there was no special need for it. 



( ) Cannot recall when I read it 



ERIC 
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5. Please rate the quality of the document in each of the following areas. If there are any outstanding^ .* 
strengths or weaknesses in an area that you feel should be considered in the preparation of future 
documents of this type, please use the comments section to explain. 





Excellent 


Good 


Fair 


Poor 


Not 

Applicable 


Choice of author 


( ) 


( ) 


( ) 


( > 


( ) 



Comments 



Selection of content/material ( ) ( J ( > ( ) ( ) 

for discussions 

Comments 



Choice of references in { ) {){)() (I 

reference list 

Comments 



Inclusion of current, up-to-date ( ) ( ) ( ) ( j ( ) 

material 

Comments ________ 



* ^ 

Accuracy in reporting facts, ( ) ( ) ( ) ( ) { ) 

events, and activities 

Comments 



2 



rN 





Excellent 


Good 


Fair 


Poor 


Not 

Applicable 


Interpretation (e.g., thought- 


( ) 


( I 


( ) 


< ) 


{ ) 



'fulness, clarity, defensibility 
and/or originality in drawing 
recommendations and conclusions) 



Comments — » 



Organization and representation • ' ( ) { j ( ) { ) ( ) 

of ideas 

Comments 



Organization of bibliographic { ) ( ) ( ) ( ) ( ) 

references 

Comments 



Format (i.e., physical layout. ( ) { ) ( ) ( ) ( ) 

illustrations, typography) 

Comments 



Clarity of writing 



( ) ( ) ( ) 



Comments 



6 . 



Would you recommend this document to your colleagues in the educational community? 



Yes ( ) No { ) 



If yes, for each of the purposes given below indicate how useful you believe it might be. 







DEGREE OF USEFULNESS 




PURPOSE OF USE 


Very 

useful 


Somewhat 

useful 


Not at 
ail useful 


To obtain an overview of 
the topic 


( ) 


( ) 


( 1 


To took up specific facts 


( ) 


( ) 


( ) 


To identify relevant 
literature references 


< ) 


( ) 


( ) 


To identify individuals 
or institutions working 
in the area 


( ) 


( ) 


{ ) 


To update knowledge about 
a known subject 


( ) 


( ) 


( ) 


To obtain knowledge about 
a new subject 


( ) 


( ) 


( ) 


To obtain practical 
guidance for educational 
practice 


( ) 


( ) 


( ) 


Other (please specify! 


( ) 


( ) 


( ) 



7. In general, how would you compare this document with other documents of the same type? 
( ) It is a very useful document. 

( ) It is not unusually useful, but it is worth having available. 

( } its usefulness is too limited to justify its publication. 

General Comments 
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APPENDIX E 



SAMPLING REPORT: RANDOM SAMPLE 



1. State Education Agencies 

2. School Districts 

3. Institutions of Higher Education 

For each of the Random Sample units, the following 
sections are included: 

a. Revised Sampling Parameters 

b. Summary of Changes and Scope Notes 

c. Sampling Report 



NOTE: Totals presented in this attachment do not 

reflect the duplications that were found 
within the sample and with the non-random 
sample. Duplications were pulled prior to 
Q1 mailings. 
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1. State Education Agencies (SEA's) 
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A. REVISED SAMPLING PARAMETERS 

STATE EDUCATION AGENCIES (SEA's) 



Sampling Categories 



Respondent 

Classification 



No. of Respondents 
Per SEA 



Total Number of 
Respondents for 8 States 



1. Deputy or Ass’t. A 

Chief State 

School Officers 

2. State School A 

Board Member 

3. Research, Planning, A 

and Evaluation Staff 

4 . Other Administrative A 

Staff 



1 



1 

2 

2 



8 



8 

16 

16 



5. Federal Programs A 1 

Coordinators/ 

Specialists 

6. Supervisors of A 

Instruction and 

Subject Area 
Consultants/ 

Specialists 



a. Elementary Education 1 

b. Special Education 1 

c. Adult Education 1 

d. Early Childhood Ed. 1 

e. Foreign Languages 1 

f . Reading 1 

g. Science 1 

h. Mathematics 1 

1. Social Studies 1 

j . English 1 

k. Vocational Education 1 

l. All others 4 

7. Library, Curriculum S 6 

Center, and Media Staff 

8. Psychologists and S 4 

Counseling/Guidance 

Staff 

Totals: 32 



8 



8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

32 

48 



32 



256 
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B. SUMMARY OF CHANGES AND SCOPE NOTES 

♦ 


Categories 


Comments 


1, 2, 3, and 5 


No substantive changes. Federal Programs' 
perscnnel (category 5) include general 
coordinators and specialists/consultants in 
specific titles (e.g., Title III, Title I) 
who are scattered throughout anggygency * s 
several divisions. w * s ’ 


4 


This category was added to pick up the 
directors and top administrators in other 
areas (e.g., facilities, budgeting, teacher 
training and accreditation) addre£»*d by the 
clearinghouses . 


6 (a~l) 


The original areas of instruction covered in 
this category were expanded to include all 
clearinghouse areas; "all others" encompasses 
areas not covered by clearinghouses. This 
change was made for several reasons: 1) super- 

visors of secondary education — an original 
category — are virtually non-existent; secondary 
education personnel are subject-specific; and 
2) any exclusion of a clearinghouse subject area 
would appear difficult to justify. 

Each category was fairly straightforward, except 
vocational education, which was defined to include 
vocational rehabilitation personnel. Typical 
"others" were physical education, driver, migrant, 
safety, fine arts, and environmental education. 


7 


This category represents a merger of two original 
categories, the reasons for which are: 1) state 
library personnel are not always a part of an SEA, 
therefore, we would have had difficulty in identi- 
fying 4 librarians; and 2) the distinction among 
instructional materials and resources personnel is 
not always as clear as our original two categories; 
therefore, placement of an individual into one or 
the other would have been arbitrary. 


8 


No changes were made except in the number sampled — 
originally we were to sample 3. The one additional 
position was taken from the total of the merged 
category (no. 7). 
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A. REVISED SAMPLING PARAMETERS 
SCHOOL DISTRICTS 



Samp link Categories Respondent 

Classification 

SCHOOL DISTRICT CENTER OFFICE 



1. Superintendent and A 

Deputy /Associate/ 

Assistant Superin- 
tendents 

2. Supervisors/ A 

Coordinators/ 

Specialists in 
Curriculum Areas 

3. Research, Planning, A 

and Evaluation Staff 

4. School Board Member A 

ELEMENTARY SCHOOL PERSONNEL 

5. Classroom and Special T 

Teachers 

6. Reading Specialist T 

7. Special Education T 

Personnel 

8. Library /Media S 

Specialists 

9. Pupil Personnel S 

Workers 

10. Principal’s Office A 

SECONDARY SCHOOL PERSONNEL 

11 . Science T 

12 . Mathematics T 

13. English/Language Arts T 

14. Foreign Languages T 

15. Social Studies/ T 

Social Sciencies 

16. Vocational Teachers T 

17. Ail Other Subjects T 

18 . Librarian/Media S 

Coordinator 

19. Counselor and S 

Psychologist 

20. Special Education S . 

21. Principal’s Office A 



TOTALS: 

o 

ERLC 



No. of 


Respondents 


Total 


No. of 


Large 


Small 


Respondents for 
8 Large 8 Small 


4 


2 


32 


16 


5 


3 


40 


24 


1 


1 


8 


8 


1 


1 


8 


8 


9 


5 


72 


40 


J 


1 


24 


8 


3 


1 


24 


8 


5 


3 


40 


24 


5 


3 


40 


24 


4 


2 


32 


16 


3 


1 


24 


8 


3 


1 


24 


8 


4 


2 


32 


16 


3 


1 


24 


8 


5 


3 


40 


24 


5 


3 


40 


24 


5 


3 


40 


24 


4 


2 


32 


16 


3 


1 


24 


8 


3 


1 


24 


8 


4 


2 


32 


16 




— - 
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82 


42 


656 


336 
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B. SUMMARY OF CHANGES AND SCOPE NOTES 



Categories 

1 



2 

3 



4 

5 



6-17 

18 

19-21 

20 



Comments 



In cases where there were only Superintendents 
(i.e., no Deputies), we sampled — when possible — 
from among Directors of major administrative units. 

To accommodate the variations in titles, we 
included supervisors specialists and coordinators. 

This category was expanded to include planning 
and evaluation staff, in addition to research 
personnel. Even so, this category represents 
the greatest hole in our sampling; very few of the 
districts had staff in this area. 

No changes. 

Special teachers (e.g., in physical education and 
music) were added to this category • In the case 
of middle schools, teaching personnel assigned to 
"clusters 1 ' or other groups similar to K-6, were 
included as elementary personnel. (Middle school 
teachers responsible for specific subject areas 
along with junior high teachers, wem considered 
secondary personnel • ) 

% 

No changes. 

In this category we merged library and media 
personnel, again because distinctions were rarely 
made in the directories. 

No changes. 

Special educators were added to make the secondary 
categories parallel in coverage of clearinghouses 
with the elementary ones. 



SAMPLING REPORT 



CO 

rH 

CO 

U 

O 

H 



CT\ 

CO 



co 



00 

CO co 



H 

<r 



CO CM 

CO CO 



00 

CO 



CM 

ox 

CM 



CM 



CM 



O tH 
CM w 



0N rH 



CM 



CM 



CM 



CM CM 



CM 



CM 



3 

4 J 

H 



00 CM 



CM CM 



CM CM 



CM 



r 



CO 

0) 

•H 

U 

O 

0C 

<tf 

4J 

c$ 

u 



/-> 

co 


co 


CO 




" 




CO 


CO 


CO 


vD O 
iH ^ 


co 


CO 




co 




CO 


CO 


CO 


/*\ 
m co 

rH w 


CO 


CO 

J 


VO 


co 




CO 


CO 


CO 






\ 


rH 


J 


\ 








^ rH 


rH 


rH 




* { 


<f 


rH 


fH 


rH 


iH w 




* 














CO CM 


CM 


CM 


S 


CM 




CM 


CM 


CM 


H w 


















CM fH 


rH 


rH 




tH 




«H 




rH 


rH s -' 














* 




iH tH 
rH w 


rH 


rH 




rH 


1 -- 


fH 


tH 


rH 


/% 
O CM 
rH *✓ 


CM 


rH 


CM 


CM 


CM 


CM 


CM 


CM 


o> co 


CM 


o 


CM 


co 


CO 


© 


rH 


rH 




















✓"N 

00 co 


CO 


CO 


CM 


CO 


CO 


O 


CO 


CM 




















r- fH 


tH 


rH 


rH 


rH 


rH 


rH 


rH 


rH 


w 


















/*> 
vD iH 


rH 


rH 


rH 


O 


r 


H 


o 


rH 


w 










1 * 








uo m 


m 


m 


in 


m 


l 


m 


uo 


m 




















/-N 

^ rH 


rH 


rH 


tH 


rH 


rH 


H 


o 


rH 


w 


















CO rH 


rH 


O 


O 


o 


rH 


O 


© 


O 




















CM CO 


CO 


CO 


o 


CM 


CO 


CM 


rH 


co 


\«/ 


















/“V 

rH CM 


rH 


CM 


rH 


CM 


CM 


H 


tH 


CM 


V/ 






















o 

ERIC 



C/3 

S 



03 

H 

Q 



00 

B 

*rl 

O 



o 

o 

® 

0 

43 

O 



O 

Js 

O m 

03 r-t 



fH <W 

S O 
•H 
O U 
*H 4J 
00 05 

ss 



g 

H 

3 



O 

a 

u 

o 

4-> 

GO 

0) 

O 

M 

s 



0) 

U 

M 

£ 



0) 

•s 

•-J 



4 

o 



43 

U 



0 

O 

U 

H 



E-9 



ii c; o 



SAMPLING REPORT 
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Total: 542 



3. Institutions of Higher Education 
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A. REVISED SAMPLING PARAMETERS 

INSTITUTIONS OF HIGHER EDUCATION 



Sampling Categories Respondent 

Classification 


No. of 
4-Year 
Univ. 


Respondents 

Jr. 

Coll. 


Total No. of 
Respondents 
4-Year Univ. Jr. 


Coll 


1 . 


President and 
Vice Presidents 


A 


1 


1 


8 


8 


2. 


Academic Deans and 
Assistant Deans 


A 


2 


1 


16 


8 


3. 


Administrative Deans 
and Administrative 
Directors 


A 


3 


2 


24 


16 


4. 


Faculty 




6 


5 


48 


40 




a. Vocational 

(Jr. Colleges) or 
Education Department 
(Universities) 


T 


(4) 


(4) 


(32) 


(32) 




b. All Others 


T 


(2) 


(1) 


(16) 


(8) 


5. 


Student Personnel Staff 


S 


3 


2 


24 


16 


6. 


Library Staff 


S 


3 


2 


24 


16 








18 


13 


144 


104 
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B. 



SUMMARY OF CHANGES AND SCOPE NOTES 



Categories Comments 

1 No changes . 

2 No changes except the addition of Assistant 

Deans. It includes those individuals who 
were listed as Dean or Assistant Dean of a 
given academic (curriculum) area, such as 
the Dean of the College of Education. Since 
some institutions used the tit la of Division 
Director rather than Dean, Directors of 
academic divisions were also included. 

o This category was added to include adminis- 
trators (e.g., in business and facilities) 

to whom some NCEC products are addressed. 

a All full time teaching staff were sampled, 

including professors, associate professors, 
assistant professors, and instructors. Part- 
time staff, such as lecturers or consultants, 
were not included. Department chairmen were 
included in this category. 

5 Deans of Students and all staff concerned with 

student affairs, including Admissions and 
Housing, were sampled in this category. 

6 Both library and media staff were included in 

this category. 
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APPENDIX F 



ADDITIONAL ANALYSES OF SCREENING QUESTIONNAIRE (Ql) RETURNS 



The following four tables display results of the analysis 
of Screening Questionnaire (Ql) respondents' interests 
("major" and "other") separately by work setting and by 
role/function. These analyses were performed prior to 
the creation of the typology of user groups that was used 
in reporting interests in Chapter IV. 
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ACRONYMS 

EDRS . ERIC Document Reproduction Sendee 
EMC . Educational Materials Center ( J isCEC) 

ERIC . Educational Resources Information Center (oi NCEC) 

GPO. U.S. Government Printing Office 

NCEC. National C -nter for Educational Communication (in USOE) 

USOE . U.S. Office of Education, Department of Health, Education and Welfare 

B. SURVEY AND ANALYSIS TERMS 

General Field Survey . Two-phase survey of a broad cross-section of educators, 
drawn from random and non-random samples. Phase 1 was a screening phase designed 
to identify potential product, evaluators. Phase 2, the Evaluation Questionnaire 
survey for product. Users and Non-users, u T as designed to obtain both evaluations 
of products that had been read, and potential-interest evaluations on products 

4 

that had not been read. 

Level-of-Ef f ort Index . An analysis variable. High, medium, and low levels 
were created for documents by product type. These v?ere developed from the 
distribution of professional m3n-hours invested in the preparation of the docu- 
ments, data for which were provided by each NCEC originating unit. 

Level-of - V: sibi.li zy Inde x. An analysis - ariabla. High, medium, and 1ou t levels 
were created for uocuments by product type. These were developed to represent 
the degree of initial exposure that a document received; in the distribution of 
original copies. (PREP reports are an exception to this criterion of "original 
distribution" as a measure of visibility. Some estimate of their secondary 
distribution through state education agencies is represented in their visibility 
index levels.) 
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Non-Random Sample . One of two samples used in the General Field Survey. Drawn 
from personnel listings of * tate education agencies, local school districts, 
junior colleges, colleges/universities, and USOE-supported research iacilities. 

Non-Reader . An evaluator- group classification for respondents to the User 
Evaluation Questionnaire (Q2) , who were Users of NCEC products (as determined 
in Ql) , but who had not read/skimmed particular products out of the 10 assigned 
documents . 

Non-User . A classification for Screening Questionnaire (Ql) respondents who 
had only heard about, or who had no awareness of NCEC products. Non-users were 
sent Non-user Evaluation Questionnaires (Q?) in the second phase of the General 
Field Survey so that their potential interest in specific products could be 
ascertained. 

Non-User Evaluation. Questionnaire (Q3) . Used in the second phase of the General 
Field Survey to obtain potential- interest information on products. 

Product Type . An analysis variable. This was the basic dimension upon which 
data were aggregated. All documents were classified as one of three product 
types: 1) Reviews and State-of-th e-Art Papers, 2) Practical Guidance Papers, 

and 3) Bibliographies. 

4 

Ql . See Screening Questionnaire. 

Q2 . See User Evaluation Questionnaire. 

Q3 . See Non-User Evaluation Questionnaire. 

Q4 . See Specialists’ Evaluation Quest ionniare. 

Random Sample . One of two samples used in the General Field Survey. Drawn from 
several listings available from dissemination channels in the "NCEC family," 
including state and local educational information center user lists, ERIC 
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clearinghouse mailing lists and request files, and ERIC Document Reproduction 
Service on-demand sales records. 



Reader . An evaluator group classification for Users in the Evaluation Question 
naire Survey who had read a particular product out of the 10 assigned. 

Screening Questionnaire (Ql) « Used in the first phase of the general Field 
Survey to identify potential product evaluators. 

Specialists 1 Evaluation Questionnaire (Q4) . Used in the Specialists’ Survey to 
obtain in-depth evaluations of products. 

Specialists’ Survey . This survey involved individuals identified by their 
colleagues as being particularly well qualified to evaluate documents ^.n their 
areas of expertise. Specialists received complete copies of products. 

Subject Area . An analysis variable. Products were classified into one of four 
broad educational areas: 1) Instructional Content, 2) Educational Administration 

and Services, 3) Special and Other Educational Groups, and 4) Higher Education. 

User . A classification for respondents to the Screening Questionnaire (Ql) 
who reported having previously read/skimmed products from at least one NCEC 
unit. Users received the User Evaluation Questionnaire and each was. asked to 
evaluate 10 products. Users w^o responded to this second questionnaire became 
Readers and/or Non-readers. 

User Evaluation Questionnaire (02) . Used in the second phase of the General 
Field Survey to obtain evaluation data on products from Readers and to obtain 
potential-interest information on products that had not been read. 

User Groups . An analysis variable. A typology of 15 general user groups was 
created from background information provided by respondents to the Screening 
Questionnaire. These groups were: Reading Specialists, Special Educators, 

Vocational Educators, Supervisors of Instruction, Counselors and Psychologists, 
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Researchers, Instructional Resources Specialists 
Assistant I’rincipals in elementary and secondary 
Secondary Teachers, College Professors, College 
in local and state education agencies, and Other 
state education agencies. 



, Program Specialists, Principal 
schools, llementary 1‘eachets, 
Administrators , Superintendents 
Administrators in local and 
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